Meta Developers

Body Tracking for Meta Quest and Meta Quest Pro uses hands and/or controller and headset movements to infer a user’s body poses. These body poses are represented as positions in a 3D space and are composed into a body tracking skeleton. Just like a a video can be composed from multiple still shots per second, repeatedly calling this API can be used to track the movements of the person wearing the headset. By mapping the joints of this skeleton onto a character rig the character can then be animated to reflect the human motions. Likewise, the position of the body can be used in game play to hit targets or to detect if the person has dodged a projectile. Note that while body poses are typically mapped to a humanoid rig, one can also map them to fantastical characters or animals.

XrBody Sample App

Build XrBody Sample App

Download the Oculus Mobile OpenXR SDK (v47 or later) and then build the XrBody sample application with:

adb uninstall com.oculus.sdk.xrbody
cd XrSamples/XrBody/Projects/Android
../../../../gradlew installDebug

Using XrBody Sample App

As a user, when you open the sample app in your ego view, you will see the skeleton joints on your body, arms, and hands drawn with the corresponding joint coordinate frames overlayed at that joint. The skeleton joints are the raw output of body tracking without any retargeting or modifications.

You can move the different joints of your body and notice from an ego view the skeleton output articulating the same way.

The Body Tracking OpenXR Extension

XR_FB_body_tracking introduces an extension to provide the output of body tracking which reconstructs the body skeleton from the three input points: the headset and both the hands/controllers. Because hand tracking and controller tracking is available on Meta Quest headsets, this extension works on these older headsets as well.

For full API reference, see Movement API reference.

Permissions

To use body tracking functionality in your application, you must add the "com.oculus.permission.BODY_TRACKING" permission to the Android manifest.

<manifest xmlns:android="http://schemas.android.com/apk/res/android">
    <!-- Body extension requirements -->
    <uses-feature android:name="com.oculus.software.body_tracking" />
    <uses-permission android:name="com.oculus.permission.BODY_TRACKING" />
        <uses-feature
            android:name="com.oculus.experimental.enabled"
            android:required="true"
            />

    ....
</manifest>

For details, read the OpenXR Support for Oculus Quest and Quest 2 guide .

Note: That’s the “install-time” permission, so it will be granted automatically. For more details about permissions, read Install-time permissions.

OpenXR Initialization

Before the app gets access to functions of a specific OpenXR extension, you must create the OpenXR session and enable the required OpenXR extension. That part of the application is common for all extensions.

During initialization, you can create the following set of objects, which will be shared between all OpenXR extensions of the app:

XrInstance instance;
XrSystemId system;
XrSession session;
XrSpace sceneSpace;

For details, see the SampleXrFramework/Src/XrApp.h header.

Process is described in OpenXR specification:

Instance: https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#instance

System: https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#system

Session: https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#session

Spaces: https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#spaces

All that initialization is implemented in SampleXrFramework/Src/XrApp.cpp.

Enabling the extension

All extensions should be explicitly listed for creating an XrInstance:

std::vector<const char*> extensions;

XrInstance instance = XR_NULL_HANDLE;

    XrInstanceCreateInfo instanceCreateInfo = {XR_TYPE_INSTANCE_CREATE_INFO};
    ....
    instanceCreateInfo.enabledExtensionCount = extensions.size();
    instanceCreateInfo.enabledExtensionNames = extensions.data();

    ....
    OXR(initResult = xrCreateInstance(&instanceCreateInfo, &instance));

For details, see SampleXrFramework/Src/XrApp.cpp.

Setting Up

The following sections provide instructions on the setting up needed to use the extension.

Include Headers

In your source code, include the following header for body tracking.

   #include <openxr/fb_body_tracking.h>

Initialize OpenXR

Prior to using body tracking, you must initialize an OpenXR session and enable the extension. For details about session initialization, read Creating Instances and Sessions.

You must initialize the OpenXR extension once and share it between all calls to the OpenXR API. If you do it successfully, you will have the following data:

    XrSession Session;
    XrSpace StageSpace;

For details, see the SampleXrFramework\Src\XrApp.h header.

It is recommended to use the constant XR_FB_BODY_TRACKING_EXTENSION_NAME as an extension name.

Check Compatibility

You must check if the user’s headset supports body tracking. For a given XrInstance, you must receive the system properties through calling the xrGetSystemProperties function to validate this. To do so, use the XrSystemBodyTrackingPropertiesFB struct that describes if a system supports body tracking. Its definition follows.

typedef struct XrSystemBodyTrackingPropertiesFB{
   XrStructureType type;
   void* XR_MAY_ALIAS next;
   XrBool32 supportsBodyTracking;
} XrSystemBodyTrackingPropertiesFB;

For details, see XrSystemBodyTrackingPropertiesFB. The following example validates body tracking support.

        XrSystemBodyTrackingPropertiesFB bodyTrackingSystemProperties{
            XR_TYPE_SYSTEM_BODY_TRACKING_PROPERTIES_FB};
        XrSystemProperties systemProperties{
            XR_TYPE_SYSTEM_PROPERTIES, &bodyTrackingSystemProperties};
        OXR(xrGetSystemProperties(GetInstance(), GetSystemId(), &systemProperties));
        if (!bodyTrackingSystemProperties.supportsBodyTracking) {
            return;
        }

For more in-depth guidance, read Collecting System Information and System Properties.

If the bodyTrackingSystemProperties field of the XrSystemBodyTrackingPropertiesFB struct returns true, body tracking is supported.

Acquire Function Pointers

To create the body tracker, you must retrieve links to all the functions in the extension before usage. For details, read GetInstance ProcTor in the OpenXR spec. Here is an example on how to achieve this.

    PFN_xrCreateBodyTrackerFB xrCreateBodyTrackerFB_ = nullptr;
    PFN_xrDestroyBodyTrackerFB xrDestroyBodyTrackerFB_ = nullptr;
    PFN_xrLocateBodyJointsFB xrLocateBodyJointsFB_ = nullptr;

        OXR(xrGetInstanceProcAddr(
            GetInstance(),
            "xrCreateBodyTrackerFB",
            (PFN_xrVoidFunction*)(&xrCreateBodyTrackerFB_)));
        OXR(xrGetInstanceProcAddr(
            GetInstance(),
            "xrDestroyBodyTrackerFB",
            (PFN_xrVoidFunction*)(&xrDestroyBodyTrackerFB_)));
        OXR(xrGetInstanceProcAddr(
            GetInstance(), "xrLocateBodyJointsFB", (PFN_xrVoidFunction*)(&xrLocateBodyJointsFB_)));

Using the Body Tracking Extension

Create Body Tracker

You can create a body tracker by using the xrCreateBodyTrackerFB function. This function creates and obtains a handle to a body tracker. Only one body tracker is allowed and multiple calls to this function will return the same handle. The handle is unique per process and cannot be shared across processes. For this call to succeed, your app must request the com.oculus.permission.BODY_TRACKING permission in the manifest, but a user is not requested to or required to grant this permission. The definition of the xrCreateBodyTrackerFB function is the following:

XrResult xrCreateBodyTrackerFB(
	XrSession session,
	const XrBodyTrackerCreateInfoFB* createInfo,
	XrBodyTrackerFB* bodyTracker);

For details, see xrCreateBodyTrackerFB. To call this function you must use an XrBodyTrackerCreateInfoFB struct which describes the requested functionality of the body tracker. The struct’s definition is as follows:

typedef struct XrBodyTrackerCreateInfoFB{
   XrStructureType type;
   const void* XR_MAY_ALIAS next;
   XrBodyJointSetFB bodyJointSet;
} XrBodyTrackerCreateInfoFB;

For details, see XrBodyTrackerCreateInfoFB. The following example demonstrates how to use these.

    XrBodyTrackerFB bodyTracker_ = XR_NULL_HANDLE;
            XrBodyTrackerCreateInfoFB createInfo{XR_TYPE_BODY_TRACKER_CREATE_INFO_FB};
            createInfo.bodyJointSet = XR_BODY_JOINT_SET_DEFAULT_FB;
            OXR(xrCreateBodyTrackerFB_(GetSession(), &createInfo, &bodyTracker_));

Retrieve Body Joint Locations

To allocate space for retrieving joint location data, you must create an XrBodyJointLocationFB struct. This is a struct that contains the estimated position, inferred orientation, and simulated tracking status of a specific body joint. It is defined as:

typedef struct XrBodyJointLocationFB{
	XrSpaceLocationFlags locationFlags;
	XrPosef pose;
} XrBodyJointLocationFB;

For details, see XrBodyJointLocationFB. Here is an example on how to create it.

    XrBodyJointLocationFB jointLocations_[XR_BODY_JOINT_COUNT_FB];

XR_BODY_JOINT_COUNT_FB is an enum that represents the number of joints in the body skeleton including the hand joints.

To retrieve joint locations each time you need them, you must use the xrLocateBodyJointsFB function. This function is used to obtain the 70 body joint locations (18 core body joints + 52 hand joints) that are tracked by the body tracker at a given point in time. Its definition follows.

XrResult xrLocateBodyJointsFB(
   XrBodyTrackerFB bodyTracker,
   const XrBodyJointsLocateInfoFB* locateInfo,
   XrBodyJointLocationsFB* locations);

See xrLocateBodyJointsFB for details. Here is an example of how to call this function. Note that ToXrTime simply converts Meta internal time in seconds to OpenXR time in nanoseconds.

            XrBodyJointsLocateInfoFB locateInfo{XR_TYPE_BODY_JOINTS_LOCATE_INFO_FB};
            locateInfo.baseSpace = GetStageSpace();
            locateInfo.time = ToXrTime(in.PredictedDisplayTime);

            XrBodyJointLocationsFB locations{XR_TYPE_BODY_JOINT_LOCATIONS_FB};
            locations.next = nullptr;
            locations.jointCount = XR_BODY_JOINT_COUNT_FB;
            locations.jointLocations = jointLocations_;
            OXR(xrLocateBodyJointsFB_(bodyTracker_, &locateInfo, &locations));

To use the data, make sure that body tracking is active and each joint location is valid, for example:

            if (locations.isActive) {
                for (int i = 0; i < XR_BODY_JOINT_COUNT_FB; ++i) {
                    if ((jointLocations_[i].locationFlags &
                            (XR_SPACE_LOCATION_ORIENTATION_VALID_BIT |
                             XR_SPACE_LOCATION_POSITION_VALID_BIT))) {
                        // the next joint location is in
                        // skeleton.joints[i].pose
                        .....
                    }
                }
            }

Retrieve T-pose Skeleton

The body skeleton is updated based on internal body skeleton calibration. When the skeleton size/proportions change, updates are indicated by incrementing the skeletonChangedCount counter, for example:

            if (locations.isActive) {
                if (locations.skeletonChangedCount != skeletonChangeCount_) {
                    skeletonChangeCount_ = locations.skeletonChangedCount;
                    // retrieve the updated skeleton
                }
            }

For allocating space to retrieve the skeleton in T-pose, use the XrBodySkeletonJointFB struct, which is a container that represents a joint in the body skeleton. An array of these structures is used to represent the joint hierarchy of the skeleton in T-pose, for example:

    XrBodySkeletonJointFB skeletonJoints_[XR_BODY_JOINT_COUNT_FB];

To retrieve the skeleton, call the xrGetSkeletonFB function which obtains the body skeleton in T-pose. By calling xrGetSkeletonFB, you query the skeleton scale and proportions in conjunction with skeletonChangedCount. The function definition follows:

XrResult xrGetSkeletonFB(
   XrBodyTrackerFB bodyTracker,
   XrBodySkeletonFB* skeleton);

For details, see xrGetSkeletonFB.

The function returns a pointer to an XrBodySkeletonFB struct. The XrBodySkeletonFB struct is a container that represents the body skeleton in T-pose including the joint hierarchy. Its definition follows:

typedef struct XrBodySkeletonFB {
   XrStructureType type;
   void* XR_MAY_ALIAS next;
   uint32_t jointCount;
   XrBodySkeletonJointFB* joints;
} XrBodySkeletonFB;

For details, see XrBodySkeletonFB.

The following example demonstrates how to use xrGetSkeletonFB.

            XrBodySkeletonFB skeleton{XR_TYPE_BODY_SKELETON_FB};
            skeleton.next = nullptr;
            skeleton.jointCount = XR_BODY_JOINT_COUNT_FB;
            skeleton.joints = skeletonJoints_;

            OXR(xrGetSkeletonFB_(bodyTracker_, &skeleton));

Note: In the returned skeleton, the joint location and orientation are relative to the parent joint and the location and orientation in space are not guaranteed. So, the skeleton is not intended for rendering and is meant for enabling retargeting. However, to visualize the skeleton by rendering it in the app, change the following flag and the T-pose skeleton will be rendered instead of tracking data.

    bool displaySkeleton_ = false;

Destroy Tracker

It is a good practice to release resources before finishing the application. To do so, use the xrDestroyBodyTrackerFB function.

            OXR(xrDestroyBodyTrackerFB_(bodyTracker_));

Did you find this page helpful?

About us

Virtual reality / Meta Horizon

Meta Horizon Creator Program

Discover

Why Meta Quest?

What is mixed reality?

Platforms and tools

2D apps for Meta Horizon OS

Support and legal

About us

Virtual reality / Meta Horizon

Meta Horizon Creator Program

Discover

Why Meta Quest?

What is mixed reality?

Platforms and tools

2D apps for Meta Horizon OS

Support and legal

About us

Virtual reality / Meta Horizon

Meta Horizon Creator Program

Discover

Why Meta Quest?

What is mixed reality?

Platforms and tools

2D apps for Meta Horizon OS

Support and legal

English (US)