Movement OpenXR Extensions uses body tracking, face tracking, and eye tracking to bring a user’s physical movements into the metaverse and enhance social experiences. Using the abstracted signals that tracking provides, developers can animate characters with social presence and provide features beyond character embodiment.

To use Movement OpenXR Extensions, the following is required:

Body Tracking for Meta Quest and Meta Quest Pro uses hands and/or controller and headset movements to infer a user’s body poses. These body poses are represented as positions in a 3D space and are composed into a body tracking skeleton. Just like a a video can be composed from multiple still shots per second, repeatedly calling this API can be used to track the movements of the person wearing the headset. By mapping the joints of this skeleton onto a character rig the character can then be animated to reflect the human motions. Likewise, the position of the body can be used in game play to hit targets or to detect if the person has dodged a projectile. Note that while body poses are typically mapped to a humanoid rig, one can also map them to fantastical characters or animals.

For more information, go to Body Tracking.

Face Tracking for Meta Quest Pro relies on inward facing cameras to detect expressive facial movements. For devices without inward facing cameras, such as Meta Quest 2 and Meta Quest 3, Face Tracking relies on audio from the microphone to estimate facial movements. These movements are categorized into expressions that are based on the Facial Action Coding System (FACS). This reference breaks down facial movements into expressions that map to common facial muscle movements like raising an eyebrow, wrinkling your nose, etc. Some common facial movements might be represented by a combination of multiple of these movements. For instance a smile could be a combination of both the right and left lip corner pullers around the mouth as well as the cheek moving or even the eyes slightly closing. For that reason, it is common to blend multiple motions together at the same time. To represent this effectively in VR or AR, the common practice is to represent these as blendshapes (also known as morph targets in some tools) with a strength that represents how strong the face is expressing this action. This API conveys each of the facial expressions as a defined blendshape with a strength that indicates the activation of that blendshape. These blendshapes can then be interpreted directly to tell if the person has their eyes open, is blinking, or smiling. They can also be combined together and retargeted to a character to provide Natural Facial Expressions.

For more information, go to Natural Facial Expressions.

Eye Tracking for Meta Quest Pro detects eye movement and enables the Eye Gaze API to drive the eye transforms for a user’s embodied character as they look around. The abstracted eye gaze representation that the API provides allows a user’s character representation to make eye contact with other users, significantly improving social presence. It can also be used to tell where the person is looking in the 3D space which can provide a good understanding of regions of interest or even be used for targeting in games.

For more information, go to Eye Tracking.