Meta Details Real-Time Avatar System Behind Muse Voice Chats

Meta detailed the real-time system behind Muse’s voice chats and animated avatars, describing a model that produces its first video frame and audio about 870 milliseconds after a user finishes speaking.

Meta launched the feature in the United States on Nov. 5 across Muse’s mobile, web and Mac apps. Users can start a session with the phone icon, interrupt Muse as it responds, and choose audio-only mode, an avatar beside the chat or a full-screen view. The app offers hundreds of voice options. Meta’s examples include planning a trip or rehearsing an interview through spoken conversation. The U.S. launch of Muse voice chats and avatars added these features to the personal AI agent.

At the system’s core, Muse Realtime Voice turns replies into voice tokens that carry both the words and tone. Muse Realtime Avatar uses the same stream to create synchronized video. Meta describes Avatar as a diffusion transformer: It generates video in short segments and uses recent frames as references for the next segment, allowing it to continue through a conversation.

Meta distilled its larger teacher model into a smaller student model to reduce generation time. The teacher needed 40 steps with three computations per step, or 120 in total; the student uses two, a 60-fold reduction. Meta says the resulting image quality is close to the teacher model’s. The avatar generates vertical video at 448 by 768 pixels and 25 frames per second. The delay from the end of a user’s turn to the first video frame and audio is about 870 milliseconds.

Meta worked with Nvidia on model optimizations, including four-bit quantization. On an Nvidia GB200, one generation step produces eight frames — 0.32 seconds of video — in 20 milliseconds. Meta says a GB200 can support 12 concurrent avatar sessions and that optimized serving capacity is eight times the baseline.

Meta also evaluated Muse against Runway Characters and HeyGen LiveAvatar using the same character for two- to three-minute conversations. Evaluators preferred Muse over Runway 78% to 22% and over HeyGen 88% to 12%. Meta said the difference in demeanor scores against Runway was not statistically significant.

Meta plans to bring voice conversations to its AI glasses and Muse Charm but has not announced a release date for either.