Foster Meaningful Connections and Interactions with Audio To Expression for Meta Horizon OS

Have you ever noticed avatars that look robotic when they talk? Rigid and uncanny mouth movements take away from the experience you’re engaging with, particularly when it diverts your attention away from the interaction at hand. This less-than-ideal representation of speaking is produced when headsets only support mouth animations and lack the ability to support visual-based face tracking and audio-based techniques.

The launch of Movement SDK brought us one step closer to tackling this issue by supporting face tracking that translates facial movements into expressions. Now, we’re going a step further with an improved version of audio-based face tracking that we call Audio To Expression in Meta Horizon OS v71. This feature enlivens and adds believability to your apps by providing more natural facial animations—even without visual-based sensors.

From the beginning, our goal has been to help you bring users closer together and become more immersed within your app experience, and a large part of this is accomplished through communication. When you communicate, a lot of the expressivity comes from the rest of your face (e.g. cheeks, eyes, eyebrows, nose), and these subtle muscle movements are missed when using traditional viseme-driven blendshapes. Audio To Expression supports these nuanced movements by using the audio stream from your app to produce full facial representations of people using only their voices.

Whether you’re building a competitive, cooperative, or casual social experience, Audio To Expression can help your experience shine. Now while users are speaking, laughing, or even coughing, their expressions will become more meaningful and indicative of their emotions.

Keep reading below to learn how Audio To Expression enriches app experiences across Meta Quest device and see how developers are leveraging this feature to make social interactions more valuable for their audience.

Audio To Expression uses advanced AI, Meta’s face tracking blendshapes (ARKit also supported), and the headset’s microphone audio stream to provide realistic full-facial animation. Best of all, it requires just a fraction of the overhead compared to the earlier Lipsync library, reducing both memory and app footprint.

This feature is available as part of the Movement SDK, which leverages the Face Tracking to enable synchronous facial representations between users and their characters or avatars. When designing characters, your character models must have blendshapes that activate the upper face. We provide a list of visual examples for each blendshape defined in the XrFaceExpression2FB enum so you can understand how various blendshapes correspond to different facial expressions.

While Audio To Expression is a game changer for producing realistic facial expressions, it isn’t limited to our newest devices. You can leverage this feature across all of our supported headsets, including Meta Quest 3s, Quest 3, Quest Pro, and Quest 2.

Audio To Expression can be leveraged across a variety of use cases, primarily ones involving social interactions between users. During our testing and evaluation process, we teamed up with Arthur Technologies to learn how this cutting-edge feature could be used in a practical way.

The platform supports enterprise collaboration in VR and beyond, helping users connect with coworkers, clients, and business partners more easily. It offers enhanced flexibility, presence, and customization compared to traditional virtual meeting services, all without the need for travel. Arthur Technologies implemented Audio to Expression with the goal of fostering empathy and trust between users through the ability to visualize the nuances of communication that are valuable in business settings.

Beyond productivity or casual social environments, Audio to Expression can make competitive experiences like first person shooter games more engaging by helping players feel more immersed in the action with their friends. Collaborative experiences also become more conducive to innovation as users have an easier time picking up on intention and meaning behind others’ words through the representation of nuanced facial movements.

VR and mixed reality experiences provide opportunities to build deeper connections than most 2D digital platforms can offer. Now with Audio To Expression, you have an easier and resource-efficient method of enabling believable and authentic representations of facial expressions to foster these types of genuine social connections. Plus, you can leverage Audio to Expression to create more pleasant and smooth interactions between users and NPCs.

The possibilities for this feature are just beginning to be uncovered, and we can’t wait to see how you leverage it to innovate and bring people closer together, no matter where they are. To get started with Audio To Expression, visit the documentation (Unity | Unreal | OpenXR).

For more of the latest developer news, be sure to follow us on X and Facebook, and don’t forget to subscribe to our monthly newsletter in your Developer Dashboard.