Meta's Muse turns any portrait into a real-time talking avatar. Meta's new audio-driven Diffusion Transformer streams 25fps avatar video at sub-second latency, beating Runway and HeyGen in head-to-head ratings. · 4 hrs ago Read 5 min * Meta launched Muse Realtime Avatar in the Muse app (US, 18+), animating reference images live during voice chats. * Audio-driven Diffusion Transformer streams 448x768 video at 25fps with ~870ms end-to-end latency. * Distillation cuts inference from 120 to 2 neural function evaluations per chunk, a 60x reduction. * Human raters preferred it over Runway Characters and HeyGen LiveAvatar across nearly every dimension. * A single NVIDIA GB200 serves 12 concurrent sessions thanks to 4-bit QAT and persistent KV caches. * All output is watermarked via Meta Video Seal to trace generated media. Meta has begun rolling out Muse Realtime Avatar to US adults, adding a visible character to the Muse app's Realtime Voice assistant. Users can supply a portrait, full-body illustration, animal, or object as a reference image. The system animates it while the assistant speaks, synchronizing lip movement, facial expression, and body gestures with the generated voice. Meta reports output at 448 x 768 pixels and 25 frames per second, with approximately 870 ms between the end of a user's turn and receipt of the first voice-and-video response data. The metric captures response startup. Meta's announcement does not provide full-response duration or results from independent network tests. One token stream aligns speech and motion. Muse Realtime Voice emits vector-quantized speech tokens, compact codes that capture both the words and their delivery. An audio decoder converts those tokens into speech, while the avatar model uses the same stream to generate matching movement. The shared representation coordinates voice, lip motion, and expression from the start of generation. The avatar generator is an audio-driven diffusion transformer conditioned on the speech tokens, reference image, and a rolling window of video latents, which are compressed representations of recent frames. It emits short causal chunks that use prior frames as motion context. Feeding each chunk into the next helps preserve the character's appearance and mannerisms while keeping memory and computation bounded during long conversations. Two steps replace 120 model passes. A conventional diffusion video model refines noise into video over many denoising steps, making real-time generation expensive. Meta's teacher model uses 40 steps with three-way classifier-free guidance, for 120 model evaluations per chunk. Classifier-free guidance combines conditioned and unconditioned predictions to steer output quality, but each additional prediction increases inference cost. The distilled student runs two unguided denoising steps and learns both the diffusion process and the effect of guidance. Meta also uses self-forcing, which trains the student on its own generated context. That exposure helps it limit drift as small errors accumulate across successive chunks. Meta reports a 60-fold reduction in computation per chunk, with human preference close to the slower teacher's result. Meta's tests put Muse ahead. In Meta's live evaluation, raters watched matched avatars conduct two- to three-minute, phone-style conversations using Muse, Runway Characters, and HeyGen LiveAvatar. The reported dimensions included lip synchronization, mannerisms, and preservation of the reference character. Meta-reported live evaluation results | Competitor | Reported finding | | Runway Characters | Raters preferred Muse on the reported dimensions except mannerisms, where the result was statistically indistinguishable from parity. | | HeyGen LiveAvatar | Raters preferred Muse on most reported dimensions, though the mannerism comparison with Runway Characters was not statistically distinguishable from parity. | The comparison reflects Meta's test design, selected workloads, and raters. Independent evaluations could establish how the results transfer across different characters, speaking styles, networks, and conversation lengths. One GB200 handles 12 sessions. Meta reports that a single NVIDIA GB200 can serve 12 concurrent real-time video-generation sessions after applying several inference and routing optimizations: * Generation speed: Each step produces eight frames, equal to 320 ms of playback, in 20 ms. That averages 2.5 ms of model time per frame. * Serving capacity: The optimized system delivers eight times the capacity of Meta's two-step BF16 baseline. BF16 is a 16-bit numeric format commonly used for model inference. * Serving stack: Persistent key-value caches, cache-aware routing, latency-aware dynamic batching, and 4-bit quantization-aware training contribute to the reported gain. The 12-session concurrency figure applies specifically to video generation. Persistent key-value caches reuse attention state from earlier chunks, while cache-aware routing sends each session to hardware that already holds its state. Latency-aware batching groups work from multiple sessions without exceeding playback deadlines. Quantization-aware training simulates 4-bit weights during fine-tuning, allowing the model to adapt to lower numerical precision before deployment. The resulting weights require less memory and data movement than the BF16 baseline. Access stays inside the Muse app. Developers cannot yet integrate the avatar into their own products because Meta has announced no public API. Current access and output are bounded as follows: * Availability: The rollout is limited to US users aged 18 and older through the Muse app. * Output: Video uses a fixed 448 x 768 portrait format at 25 frames per second, targeting phone-sized interactions rather than larger production formats. * Provenance: Every generated video carries a durable, invisible watermark embedded with Meta Video Seal. Meta says watermarking adds no latency to generation. For voice-agent developers, Muse provides a concrete architecture for synchronizing generated speech and video with sub-second reported startup latency. Shared speech tokens coordinate the modalities, causal latent windows bound the model's context, and distillation plus low-precision serving reduce inference cost. Open questions include performance under independent testing, efficiency on accelerators other than the GB200, and whether Meta will expose the system through an API.

Artificial Analysis' Code2Video-AA benchmark shows Claude Opus 5.5 dominates AI video coding. A new benchmark judges 27 models on turning marketing briefs into animated HTML videos, with Claude Opus 5.5 leading at 10x the cost of rivals. · 3 hrs ago Read 5 min * Artificial Analysis launched Code2Video-AA, benchmarking 27 models on 114 marketing video briefs rendered from HTML. * Claude Opus 5.5 leads at 1536 Elo, winning all five judge axes including craft and engagement. * Cost per task spans 300x, from $0.01 for MiMo-V2.6-Flash to $3.89 for Claude Fable 5.1. * GPT-6.1 Sol delivers 1369 Elo at $0.31 per task, roughly 10% of Opus's cost. * Open weights models trail the frontier by nearly 500 Elo points on this visual design workload. * Over 35,000 pairwise verdicts from a Gemini 4 Argon judge produced the Elo rankings with 95% confidence intervals. Artificial Analysis has released Code2Video-AA, an evaluation developed with HeyGen that tests whether language models can translate marketing briefs into animated videos using code alone. Conventional coding and reasoning benchmarks reveal little about layout, motion, timing, and visual polish. This benchmark gives teams building code-driven video products a shared comparison of output quality, render reliability, and generation cost. Artificial Analysis runs Code2Video-AA as an independent implementation of HeyGen's Code2Video Bench. Across 114 briefs, each of 27 models creates a HyperFrames composition in a single HTML file. HTML defines the layout, GSAP controls the animation, and HyperFrames renders the result as video. One file, one render. * Prompt: The evaluator sends a design brief and HeyGen's system prompt in one model call, with tools disabled. * Generation: The model returns a single index.html file containing the complete composition. * Rendering: HyperFrames renders the file offline with no network access. Missing CDN assets, unavailable scripts, and invalid code can cause the submission to fail. * Judging: A failed render loses every comparison against a successfully rendered video for the same brief. Gemini 4 Argon compares pairs of rendered videos without seeing the model names. It selects a winner on five axes: engagement, prompt intent, composition, temporal consistency, and craft. A Bradley-Terry model converts those pairwise verdicts into an Elo rating for each axis, and the benchmark averages the five ratings for its overall score. GPT-5.5 anchors the scale at 1,000, so every Elo score is relative to performance within this evaluation. Claude takes all five axes. Claude Opus 5.5 leads with 1,536 Elo and ranks first on every judging axis. Claude Sonnet 5.5 follows at 1,443, while GPT-6 Astra scores 1,427; the benchmark reports those two models as statistically tied for second. GPT-6.1 Sol places fourth at 1,369. Leading overall scores | Model | Elo | Result | | Claude Opus 5.5 | 1,536 | First on all five axes | | Claude Sonnet 5.5 | 1,443 | Statistically tied for second | | GPT-6 Astra | 1,427 | Statistically tied for second | | GPT-6.1 Sol | 1,369 | Fourth overall | Opus records its largest lead on craft, where it finishes 128 points above Sonnet. Its narrowest lead is 48 points on prompt intent. The leading open-weight models remain nearly 500 points behind Opus: DeepSeek V4.1 Flash scores 1,053, while MiMo-V2.6-Flash and MiMo-V2.6-Pro each score 1,052. Costs span more than 300-fold. Reported cost per task ranges from $0.01 for MiMo-V2.6-Flash to $3.89 for Claude Fable 5.1. The same benchmark workload therefore carries a price spread exceeding 300-fold across providers and models. Eight models sit on the Elo-to-cost Pareto frontier, meaning no alternative in the evaluation offers both a higher score and a lower reported cost. Claude Haiku 5.5 reaches 1,192 Elo at $0.08 per task, while GPT-6.1 Sol reaches 1,369 at $0.31. GPT-6 Astra gains 58 Elo over Sol but costs $1.62 per task, about 5.2 times as much. Reasoning tokens account for most of the variation in output usage. Average output ranges from about 11,000 tokens per task for Nemotron 3 Ultra to about 176,000 for Claude Sonnet 5.5, while the final HTML compositions occupy roughly 4,000 to 12,000 tokens. The three Claude 5.5 models use approximately 152,000 to 176,000 output tokens per task, compared with a median of about 34,000 across the other 24 models. One score bundles three skills. A high Code2Video-AA score requires models to satisfy dense instructions, generate executable front-end code, and make coherent visual choices within one response. * Long-form instruction following: Briefs specify details such as hex colors, scene order, animation timing, and camera behavior. * Front-end code generation: Models must produce a working GSAP animation in one HTML file without a debugging loop. * Visual judgment: Engagement and craft reward hierarchy, pacing, polish, and other qualities that extend beyond technical correctness. Rendering failures explain little of the separation among the strongest models. Of 3,078 submissions, 3,042 rendered successfully, a rate of 98.8%. Eight models completed all 114 briefs, and the lowest result was GLM-5.3 with 110 successful renders. Among models with near-perfect reliability, Elo differences mainly reflect the judge model's visual preferences. A narrow slice of visual development. Code2Video-AA measures a specific workflow: single-call generation of marketing animations in HyperFrames. Several boundaries affect how its results should be applied. * No iteration: The setup excludes tool use, debugging, human feedback, and multi-step agent workflows. * One rendering stack: Results cover HTML and GSAP compositions rendered through HyperFrames. Other animation frameworks and video-editing systems may produce different rankings. * One automated judge: Gemini 4 Argon supplies every aesthetic verdict, so its stylistic preferences influence the Elo ratings. * Relative ratings: Elo supports comparisons among models in this benchmark. It does not provide an absolute percentage of visual quality or production readiness. Sol narrows the gap at $0.31. Teams building code-driven motion products can use the Pareto frontier to identify candidates for further testing. GPT-6.1 Sol trails Opus by 167 Elo while costing $0.31 per task, and Claude Haiku 5.5 offers a lower-cost option at $0.08. GPT-6 Astra provides a smaller score gain over Sol at a substantially higher reported price. Opus remains the strongest candidate when judge-rated craft is the primary selection criterion. A production evaluation should also cover representative brand briefs, browser compatibility, generation latency, editing requirements, and failure recovery. Those factors fall outside Code2Video-AA but can materially change the cost of deploying a model. The full leaderboard includes model-level results and rendered samples. Artificial Analysis also publishes its methodology, while HeyGen provides the HyperFrames renderer on GitHub for local reproduction.