Meta Developers

For applications running on the device, face tracking data is protected by the permission com.oculus.permission.FACE_TRACKING, com.oculus.permission.EYE_TRACKING, and android.permission.RECORD_AUDIO. Without com.oculus.permission.FACE_TRACKING your application cannot access this API. Without com.oculus.permission.EYE_TRACKING, the eye-look blendshapes will not be populated. android.permission.RECORD_AUDIO is required to use audio-driven face tracking, especially for the devices that don’t have inward facing cameras. See Android Permissions below for instructions on how to declare and request this permission in your application.

Face Tracking Functions

XrResult xrCreateFaceTracker2FB(
   XrSession session,
   const XrFaceTrackerCreateInfo2FB* createInfo,
   XrFaceTracker2FB* faceTracker);

xrCreateFaceTracker2FB is used to create and obtain a XrFaceTracker2FB handle to a Facebook Face Tracker. Only one instance of the Facebook Face Tracker is allowed per process and multiple calls to this function will return the same handle. The handle is unique per process. For this call to succeed, applications must request the com.oculus.permission.FACE_TRACKING permission in their manifest, and a user must grant this permission. Face tracking blendshape data will be available via the xrGetFaceExpressionWeights2FB immediately upon return of this call.

The following are the parameters for this function call:

XrSession session: A handle to the XR session.

XrFaceTrackerCreateInfo2FB* createInfo: A structure used to describe the capabilities requested to create the face tracker.

XrFaceTracker2FB* faceTracker: Upon completion with XR_SUCCESS, this will be a valid handle to the Facebook Face Tracker supporting requested capabilities.

XrResult xrDestroyFaceTracker2FB(
   XrFaceTracker2FB faceTracker);

xrDestroyFaceTracker2FB is used to destroy the Facebook Face Tracker and release its resources. The following are the parameters for this function call:

XrFaceTracker2FB faceTracker: The handle obtained on a call to xrCreateFaceTracker2FB. Upon completion of xrDestroyFaceTracker2FB The handle is no longer valid.

XrResult XRAPI_CALL xrGetFaceExpressionWeights2FB(
   XrFaceTracker2FB faceTracker,
   const XrFaceExpressionInfo2FB* expressionInfo,
   XrFaceExpressionWeights2FB* expressionWeights);

xrGetFaceExpressionWeights2FB is used to obtain the weights and confidences for the 63 blendshapes that are tracked by the Facebook Face Tracker at a given point in time. The following are the parameters for this function call:

XrFaceTracker2FB faceTracker: A handle to a face tracker created on successful call to xrCreateFaceTracker2FB.

const XrFaceExpressionInfo2FB* expressionInfo: A structure containing the required time at which weights and confidences for the blendshapes is returned. Callers should request an XrTime that is equal to the predicted display time for the rendered frame.

XrFaceExpressionWeights2FB* expressionWeights: The XrFaceExpressionWeights2FB data structure containing an array of expression weights, confidences and the status reporting if the blendshape weights are valid. The elements in weights are ordered according to the XrFaceExpression2FB enum and the elements in confidences are ordered according to the XrFaceConfidence2FB enum. Callers must ensure weights array is of size XR_FACE_EXPRESSION2_COUNT_FB, that weightsCount is equal to this size and ensure confidences array is of size XR_FACE_CONFIDENCE2_COUNT_FB, that confidenceCount is equal to this size.

Face Tracking Data Structures

typedef struct XrSystemFaceTrackingProperties2FB {
   XrStructureType    type;
   void* XR_MAY_ALIAS next;
   XrBool32           supportsVisualFaceTracking;
   XrBool32           supportsAudioFaceTracking;
} XrSystemFaceTrackingProperties2FB;

XrSystemFaceTrackingProperties2FB is a structure used to describe if a system supports face tracking. It is obtained through a call to xrGetSystemProperties() It has the following members:

XrStructureType type: The XrStructureType of this structure.

void* XR_MAY_ALIAS next: NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR for this extension.

XrBool32 supportsVisualFaceTracking: A boolean indicating if face tracking is supported by the device with inward facing cameras. TRUE indicates support while FALSE indicates no support.

XrBool32 supportsAudioFaceTracking: A boolean indicating if audio-driven face tracking is supported by the device/system. TRUE indicates support while FALSE indicates no support.

A call to xrCreateFaceTracker2FB will fail with a failure XrResult of XR_ERROR_FEATURE_UNSUPPORTED if the system does not support any types of face tracking.

typedef enum XrFaceExpressionSet2FB {
    XR_FACE_EXPRESSION_SET2_DEFAULT_FB = 0,
    XR_FACE_EXPRESSION_SET_2FB_MAX_ENUM_FB = 0x7FFFFFFF
} XrFaceExpressionSet2FB;

XrFaceExpressionSet2FB is an enum used to describe the requested set of expressions returned by the face tracker. It has the following possible values:

XR_FACE_EXPRESSION_SET2_DEFAULT_FB: The default set of expressions.

typedef struct XrFaceTrackerCreateInfo2FB {
    XrStructureType              type;
    const void* XR_MAY_ALIAS     next;
    XrFaceExpressionSet2FB       faceExpressionSet;
    uint32_t                     requestedDataSourceCount;
    XrFaceTrackingDataSource2FB* requestedDataSources;
} XrFaceTrackerCreateInfo2FB;

XrFaceTrackerCreateInfo2FB is a structure used to describe the requested functionality of the face tracker. It has the following members:

XrStructureType type: The XrStructureType of this structure.

const void* XR_MAY_ALIAS next: NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

XrFaceExpressionSet2FB faceExpressionSet: The requested capability flags. See XrFaceExpressionSet2FB for more information.

uint32_t requestedDataSourceCount: The number of elements in the requestedDataSources array.

XrFaceTrackingDataSource2FB* requestedDataSources: An array of XrFaceTrackingDataSource2FB that the application accepts. For example, if you want to use face tracking only for the devices that have inward facing cameras, you can pass an array with a single element of XR_FACE_TRACKING_DATA_SOURCE2_VISUAL_FB. If you pass size two array with XR_FACE_TRACKING_DATA_SOURCE2_VISUAL_FB and XR_FACE_TRACKING_DATA_SOURCE2_AUDIO_FB, the system will choose any supported data source, preferably one is more expressive than the others. You can also choose to pass an empty array and 0 to requestedDataSourceCount to achieve the same purpose of letting the system decide.

typedef enum XrFaceTrackingDataSource2FB {
    XR_FACE_TRACKING_DATA_SOURCE2_VISUAL_FB = 0,
    XR_FACE_TRACKING_DATA_SOURCE2_AUDIO_FB = 1,
    XR_FACE_TRACKING_DATA_SOURCE_2FB_MAX_ENUM_FB = 0x7FFFFFFF
} XrFaceTrackingDataSource2FB;

XrFaceTrackingDataSource2FB is an enum used to describe the requested set of data sources that can be used by face tracker. It has the following possible values:

XR_FACE_TRACKING_DATA_SOURCE2_VISUAL_FB: This value indicates that the face tracking data source supports using inward facing camera data to estimate facial expression. The system may also use audio to further improve the quality of the tracking.

XR_FACE_TRACKING_DATA_SOURCE2_AUDIO_FB: This value indicates that the face tracking data source supports using audio data to estimate facial expression. The runtime must not use camera data for this data source.

typedef enum XrFaceConfidence2FB {
    XR_FACE_CONFIDENCE2_LOWER_FACE_FB = 0,
    XR_FACE_CONFIDENCE2_UPPER_FACE_FB = 1,
    XR_FACE_CONFIDENCE2_COUNT_FB = 2,
    XR_FACE_CONFIDENCE_2FB_MAX_ENUM_FB = 0x7FFFFFFF
} XrFaceConfidence2FB;

XrFaceConfidence2FB is an enum used to describe the confidence in the upper and lower face regions. XR_FACE_CONFIDENCE2_COUNT_FB defines the number of confidences returned within XrFaceExpressionWeights2FB.

typedef struct XrFaceExpressionInfo2FB {
    XrStructureType          type;
    const void* XR_MAY_ALIAS next;
    XrTime                   time;
} XrFaceExpressionInfo2FB;

XrFaceExpressionInfo2FB is a structure used to describe the time at which face expressions are being requested. Callers should request a time equal to the predicted display time for the rendered frame. The system will return the value at the closest timestamp possible to the requested timestamp. The timestamp of the estimation is always provided so that the caller can determine to the extent the system was able to fulfill the request. The system will employ appropriate modeling to provide expressions for this time. It has the following members:

XrStructureType type: The XrStructureType of this structure.

const void* XR_MAY_ALIAS next: NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

XrTime time: The time at which the face expression weights and confidences are requested. Callers should request a time equal to the predicted display time for the rendered frame. The system will employ appropriate modeling to provide expressions for this time.

typedef struct XrFaceExpressionWeights2FB {
    XrStructureType             type;
    void* XR_MAY_ALIAS          next;
    uint32_t                    weightCount;
    float*                      weights;
    uint32_t                    confidenceCount;
    float*                      confidences;
    XrBool32                    isValid;
    XrBool32                    isEyeFollowingBlendshapesValid;
    XrFaceTrackingDataSource2FB dataSource;
    XrTime                      time;
} XrFaceExpressionWeights2FB;

XrFaceExpressionWeights2FB is a structure that contains arrays describing the face tracking blendshape weights and confidences. It has the following members:

XrStructureType type: The XrStructureType of this structure.

void* XR_MAY_ALIAS next: NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

uint32_t weightCount: The size of the weights arrays. If the weightCount is not equal to XR_FACE_EXPRESSION_COUNT_FB The system will return XR_ERROR_VALIDATION_FAILURE.

float* weights: An array of floats of size weightCount with values between 0 and 1. The order and values correspond to weights for each of the face tracking blendshapes defined in XrFaceExpressionFB. A value of 0 indicates no application of the corresponding blendshape, and a value of 1 indicates maximum application of the corresponding blendshape.

uint32_t confidenceCount: The size of the confidences arrays. If the confidenceCount is not equal to XR_FACE_CONFIDENCE_COUNT_FB the system will return XR_ERROR_VALIDATION_FAILURE.

float* confidences: An array of floats of size XR_FACE_CONFIDENCE_COUNT_FB valued between 0 and 1. The order and values correspond to confidence for each of the face tracking blendshapes defined in XrFaceConfidenceFB. A value of 0 indicates no confidence for the blendshape weights in the corresponding face region, and a value of 1 indicates maximum confidence for the blendshape weights in the corresponding face region.

XrBool32 isValid: A bool to indicate if the returned blendshapes are valid. Callers should check the validity of blendshapes prior to use. Temporary failures such as lack of permissions or lack of application focus will cause isValid to be FALSE.

XrBool32 isEyeFollowingBlendshapesValid: A bool to indicate if the 8 expression weights with prefix XR_FACE_EXPRESSION_EYES_LOOK_* are valid. If the permission for eye tracking is not granted for this app then the 8 expression weights are invalid.

XrFaceTrackingDataSource2FB dataSource: An XrFaceTrackingDataSource2FB enum, which indicates the data source that was used to estimate the facial expression.

XrTime time: Upon return, the value will be set to the time at which the expressions were tracked. This time may not be the same as the requested time if the system was unable to extrapolate expressions to the requested time.

typedef enum XrFaceExpression2FB {
    XR_FACE_EXPRESSION2_BROW_LOWERER_L_FB = 0,
    XR_FACE_EXPRESSION2_BROW_LOWERER_R_FB = 1,
    XR_FACE_EXPRESSION2_CHEEK_PUFF_L_FB = 2,
    XR_FACE_EXPRESSION2_CHEEK_PUFF_R_FB = 3,
    XR_FACE_EXPRESSION2_CHEEK_RAISER_L_FB = 4,
    XR_FACE_EXPRESSION2_CHEEK_RAISER_R_FB = 5,
    XR_FACE_EXPRESSION2_CHEEK_SUCK_L_FB = 6,
    XR_FACE_EXPRESSION2_CHEEK_SUCK_R_FB = 7,
    XR_FACE_EXPRESSION2_CHIN_RAISER_B_FB = 8,
    XR_FACE_EXPRESSION2_CHIN_RAISER_T_FB = 9,
    XR_FACE_EXPRESSION2_DIMPLER_L_FB = 10,
    XR_FACE_EXPRESSION2_DIMPLER_R_FB = 11,
    XR_FACE_EXPRESSION2_EYES_CLOSED_L_FB = 12,
    XR_FACE_EXPRESSION2_EYES_CLOSED_R_FB = 13,
    XR_FACE_EXPRESSION2_EYES_LOOK_DOWN_L_FB = 14,
    XR_FACE_EXPRESSION2_EYES_LOOK_DOWN_R_FB = 15,
    XR_FACE_EXPRESSION2_EYES_LOOK_LEFT_L_FB = 16,
    XR_FACE_EXPRESSION2_EYES_LOOK_LEFT_R_FB = 17,
    XR_FACE_EXPRESSION2_EYES_LOOK_RIGHT_L_FB = 18,
    XR_FACE_EXPRESSION2_EYES_LOOK_RIGHT_R_FB = 19,
    XR_FACE_EXPRESSION2_EYES_LOOK_UP_L_FB = 20,
    XR_FACE_EXPRESSION2_EYES_LOOK_UP_R_FB = 21,
    XR_FACE_EXPRESSION2_INNER_BROW_RAISER_L_FB = 22,
    XR_FACE_EXPRESSION2_INNER_BROW_RAISER_R_FB = 23,
    XR_FACE_EXPRESSION2_JAW_DROP_FB = 24,
    XR_FACE_EXPRESSION2_JAW_SIDEWAYS_LEFT_FB = 25,
    XR_FACE_EXPRESSION2_JAW_SIDEWAYS_RIGHT_FB = 26,
    XR_FACE_EXPRESSION2_JAW_THRUST_FB = 27,
    XR_FACE_EXPRESSION2_LID_TIGHTENER_L_FB = 28,
    XR_FACE_EXPRESSION2_LID_TIGHTENER_R_FB = 29,
    XR_FACE_EXPRESSION2_LIP_CORNER_DEPRESSOR_L_FB = 30,
    XR_FACE_EXPRESSION2_LIP_CORNER_DEPRESSOR_R_FB = 31,
    XR_FACE_EXPRESSION2_LIP_CORNER_PULLER_L_FB = 32,
    XR_FACE_EXPRESSION2_LIP_CORNER_PULLER_R_FB = 33,
    XR_FACE_EXPRESSION2_LIP_FUNNELER_LB_FB = 34,
    XR_FACE_EXPRESSION2_LIP_FUNNELER_LT_FB = 35,
    XR_FACE_EXPRESSION2_LIP_FUNNELER_RB_FB = 36,
    XR_FACE_EXPRESSION2_LIP_FUNNELER_RT_FB = 37,
    XR_FACE_EXPRESSION2_LIP_PRESSOR_L_FB = 38,
    XR_FACE_EXPRESSION2_LIP_PRESSOR_R_FB = 39,
    XR_FACE_EXPRESSION2_LIP_PUCKER_L_FB = 40,
    XR_FACE_EXPRESSION2_LIP_PUCKER_R_FB = 41,
    XR_FACE_EXPRESSION2_LIP_STRETCHER_L_FB = 42,
    XR_FACE_EXPRESSION2_LIP_STRETCHER_R_FB = 43,
    XR_FACE_EXPRESSION2_LIP_SUCK_LB_FB = 44,
    XR_FACE_EXPRESSION2_LIP_SUCK_LT_FB = 45,
    XR_FACE_EXPRESSION2_LIP_SUCK_RB_FB = 46,
    XR_FACE_EXPRESSION2_LIP_SUCK_RT_FB = 47,
    XR_FACE_EXPRESSION2_LIP_TIGHTENER_L_FB = 48,
    XR_FACE_EXPRESSION2_LIP_TIGHTENER_R_FB = 49,
    XR_FACE_EXPRESSION2_LIPS_TOWARD_FB = 50,
    XR_FACE_EXPRESSION2_LOWER_LIP_DEPRESSOR_L_FB = 51,
    XR_FACE_EXPRESSION2_LOWER_LIP_DEPRESSOR_R_FB = 52,
    XR_FACE_EXPRESSION2_MOUTH_LEFT_FB = 53,
    XR_FACE_EXPRESSION2_MOUTH_RIGHT_FB = 54,
    XR_FACE_EXPRESSION2_NOSE_WRINKLER_L_FB = 55,
    XR_FACE_EXPRESSION2_NOSE_WRINKLER_R_FB = 56,
    XR_FACE_EXPRESSION2_OUTER_BROW_RAISER_L_FB = 57,
    XR_FACE_EXPRESSION2_OUTER_BROW_RAISER_R_FB = 58,
    XR_FACE_EXPRESSION2_UPPER_LID_RAISER_L_FB = 59,
    XR_FACE_EXPRESSION2_UPPER_LID_RAISER_R_FB = 60,
    XR_FACE_EXPRESSION2_UPPER_LIP_RAISER_L_FB = 61,
    XR_FACE_EXPRESSION2_UPPER_LIP_RAISER_R_FB = 62,
    XR_FACE_EXPRESSION2_TONGUE_TIP_INTERDENTAL_FB = 63,
    XR_FACE_EXPRESSION2_TONGUE_TIP_ALVEOLAR_FB = 64,
    XR_FACE_EXPRESSION2_TONGUE_FRONT_DORSAL_PALATE_FB = 65,
    XR_FACE_EXPRESSION2_TONGUE_MID_DORSAL_PALATE_FB = 66,
    XR_FACE_EXPRESSION2_TONGUE_BACK_DORSAL_VELAR_FB = 67,
    XR_FACE_EXPRESSION2_TONGUE_OUT_FB = 68,
    XR_FACE_EXPRESSION2_TONGUE_RETREAT_FB = 69,
    XR_FACE_EXPRESSION2_COUNT_FB = 70,
    XR_FACE_EXPRESSION_2FB_MAX_ENUM_FB = 0x7FFFFFFF
} XrFaceExpression2FB;

The structure that describes the blendshapes supported by the face tracker.

Facebook Eye Tracking OpenXR API

Eye Tracking Permission

For applications running on the device, eye tracking data is protected by the permission com.oculus.permission.EYE_TRACKING. Without this permission, your application cannot access this API. See Android Permissions below for instructions on how to declare and request this permission in your application.

Eye Tracking Functions

XrResult xrCreateEyeTrackerFB(
    XrSession session,
    const XrEyeTrackerCreateInfoFB* createInfo,
    XrEyeTrackerFB* eyeTracker);

xrCreateEyeTrackerFB is used to create and obtain a XrEyeTrackerFB handle to a Facebook Eye Tracker. Only one Facebook Eye Tracker is allowed and multiple calls to this function will return the same handle. The handle is unique per process and cannot be shared across processes. For this call to succeed, applications must request the com.oculus.permission.EYE_TRACKING permission in their manifest, and a user must grant this permission. Eye tracking gaze data will be available immediately through the xrGetGazesFB function call upon completion of the call to xrCreateEyeTrackerFB. The following are the parameters for this function call:

XrSession session: A handle to the XR session.

XrEyeTrackerCreateInfoFB* createInfo: A structure used to describe the capabilities requested to create the eye tracker.

XrEyeTrackerFB* eyeTracker: Upon completion with XR_SUCCESS, this will be a valid handle to the Facebook Eye Tracker supporting requested capabilities.

XrResult xrDestroyEyeTrackerFB(
   XrEyeTrackerFB eyeTracker);

xrDestroyEyeTrackerFB is used to destroy the Facebook Eye Tracker and release its resources. The following are the parameters for this function call:

XrEyeTrackerFB eyeTracker: The handle obtained on a call to xrCreateEyeTrackerFB. Upon completion of xrDestroyEyeTrackerFB the handle is no longer valid.

XrResult xrGetEyeGazesFB(
   XrEyeTrackerFB eyeTracker,
   const XrEyeGazesInfoFB* gazeInfo,
   XrEyeGazesFB* eyeGazes);

xrGetEyeGazeFB is used to obtain pose for a user’s eyes at a specific time and within a specific coordinate system. The following are the parameters for the function call:

XrEyeTrackerFB eyeTracker: A handle to the XrEyeTrackerFB created by calling xrCreateEyeTrackerFB.

const XrEyeGazesInfoFB* gazeInfo: The XrGazesInfoFB structure that contains the requested time and space for the required eye gaze positions. Callers should request a time equal to the predicted display time for the rendered frame. The system will employ appropriate modeling to provide eye gaze for this time.

XrEyeGazesFB* eyeGazes: The XrEyeGazesFB structure that contains an array of returned eye poses and confidence.

Eye Tracking Data Structures

typedef enum XrEyePositionFB {
    XR_EYE_INDEX_LEFT_FB = 0,
    XR_EYE_INDEX_RIGHT_FB = 1,
    XR_EYE_INDEX_COUNT_FB = 2,
    XR_EYE_POSITION_MAX_ENUM_FB = 0x7FFFFFFF
} XrEyePositionFB;

XrEyePositionFB is an enum used to describe the index ordering for the returned XrEyeGazesFB array. This enum has the following values:

XrStructureType type: The XrStructureType of this structure.

XR_EYE_INDEX_LEFT_FB: The index value representing the left eye gaze information.

XR_EYE_INDEX_RIGHT_FB: The index value representing the right eye gaze information.

XR_EYE_INDEX_COUNT_FB: The expected number of different XrEyeGazeFB returned. The value is two, one for the left eye and one for the right eye.

XR_EYE_POSITION_MAX_ENUM_FB: For future use. The maximum number of indexes to be supported.

typedef struct XrSystemEyeTrackingPropertiesFB {
   XrStructureType type;
   void* XR_MAY_ALIAS next;
   XrBool32 supportsEyeTracking;
} XrSystemEyeTrackingPropertiesFB;

XrSystemEyeTrackingPropertiesFB is a structure obtained through a call to xrGetSystemProperties used to describe the system’s support for eye tracking. It has the following members:

XrStructureType type: The XrStructureType of this structure.

void* XR_MAY_ALIAS next: NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

XrBool32 supportsEyeTracking: A boolean indicating if the eye tracking is supported by the device/system and the application has declared the usage of this feature by requesting permission com.oculus.permission.EYE_TRACKING in the manifest. TRUE if the system supports eye tracking, FALSE if the system does not. A call to xrCreateEyeTrackerFB will fail with a failure XrResult of XR_ERROR_FEATURE_UNSUPPORTED if the system does not support eye tracking.

typedef struct XrEyeTrackerCreateInfoFB {
   XrStructureType type;
   const void* XR_MAY_ALIAS next;
} XrEyeTrackerCreateInfoFB;

XrEyeTrackerCreateInfoFB is a structure used to describe the requested functionality of the eye tracker. It has the following members:

XrStructureType type: The XrStructureType of this structure.

const void* XR_MAY_ALIAS next: NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

typedef struct XrEyeGazesInfoFB {
   XrStructureType type;
   const void* XR_MAY_ALIAS next;
   XrSpace baseSpace;
   XrTime time;
} XrEyeGazesInfoFB;

XrEyeGazesInfoFB is a structure used to describe the time at which eye gaze is being requested. Callers should request a time equal to the predicted display time for the rendered frame. The system will employ appropriate modeling to provide eye gaze for this time. It has the following members:

XrStructureType type: The XrStructureType of this structure.

const void* XR_MAY_ALIAS next: NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

XrSpace baseSpace: The XrSpace within which the returned eye poses will be represented.

XrTime time: The time at which the eye gaze information is requested. Callers should request a time equal to the predicted display time for the rendered frame. The system will employ appropriate modeling to provide eye gaze for this time.

typedef struct XrEyeGazeFB {
    XrBool32 isValid;
    XrPosef gazePose;
    float gazeConfidence;
} XrEyeGazeFB;

XrEyeGazeFB is a structure that contains gaze pose representing the user’s eye location and gaze direction along with a confidence of these measurements. It has the following members:

XrBool32 isValid: A bool to indicate if the returned gazePose is valid. Callers should check the validity of pose prior to use. Temporary failures such as lack of permissions or loss of application focus will cause isValid to be FALSE.

XrPosef gazePose: An XrPosef pose describing the position and orientation of the user’s eye. The pose is represented in the coordinate system provided by baseSpace in the XrEyeGazeInfoFB structure.

float gazeConfidence: A float between 0 and 1 which represents the confidence for eye pose. A value of 0 means no confidence in the pose returned, and a value of 1 means maximum confidence in the returned eye pose.

typedef struct XrEyeGazesFB {
   XrStructureType type;
   void* XR_MAY_ALIAS next;
   XrEyeGazeFB gaze[XR_EYE_INDEX_COUNT_FB];
   XrTime time;
} XrEyeGazesFB;

XrEyeGazesFB is a structure that contains an array of gaze pose and confidence per eye and return timestamp. It has the following members:

XrStructureType type: The XrStructureType of this structure.

void* XR_MAY_ALIAS next: NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

XrEyeGazeFB gaze[XR_EYE_INDEX_COUNT_FB]: An array of XrEyeGazeFB containing pose and confidence for both the user’s eyes. Index 0 represents the user’s left eye, and index 1 represents the user’s right eye.

XrTime time: Upon return, the value will be set to the time at which the gazes were tracked. This time may not be the same as the requested time if the system was unable to extrapolate gazes to the requested time.

Facebook Body Tracking OpenXR API

Body Tracking Functions

XrResult xrCreateBodyTrackerFB(
    XrSession session,
    const XrBodyTrackerCreateInfoFB* createInfo,
    XrBodyTrackerFB* bodyTracker);

xrCreateBodyTrackerFB is used to create and obtain a handle to a Facebook Body Tracker. Only one Facebook Body Tracker is allowed and multiple calls to this function will return the same handle. The handle is unique per process and cannot be shared across processes. For this call to succeed, applications must request the com.oculus.permission.BODY_TRACKING permission in their manifest, but a user is not requested to or required to grant this permission. The following are the parameters for this function call:

XrSession session: A handle to the XR session.

XrBodyTrackerCreateInfoFB* createInfo: A structure used to describe the capabilities requested to create the body tracker.

XrBodyTrackerFB* bodyTracker: Upon completion with XR_SUCCESS, this will be a valid handle to the Facebook Body Tracker supporting requested capabilities.

XrResult xrDestroyBodyTrackerFB(
   XrBodyTrackerFB bodyTracker);

xrDestroyBodyTrackerFB is used to destroy the Facebook Body Tracker and release its resources. The following are the parameters for this function call:

XrBodyTrackerFBbodyTracker: The handle obtained on a call to xrCreateBodyTrackerFB. Upon completion of xrDestroyBodyTrackerFB, the handle is no longer valid.

XrResult xrGetSkeletonFB(
   XrBodyTrackerFB bodyTracker,
   XrBodySkeletonFB* skeleton);

xrGetSkeletonFB is used to obtain the body skeleton in T-pose. This function can be used to query the skeleton scale and proportions in conjunction with skeletonChangedCount. The following are the parameters for this function call:

XrBodyTrackerFBbodyTracker: The handle obtained on a call to xrCreateBodyTrackerFB.

XrBodySkeletonFB skeleton: The returned skeleton.

XrResult xrLocateBodyJointsFB(
   XrBodyTrackerFB bodyTracker,
   const XrBodyJointsLocateInfoFB* locateInfo,
   XrBodyJointLocationsFB* locations);

xrLocateBodyJointsFB is used to obtain the 70 body joint locations (18 core body joints + 52 hand joints) that are tracked by the Facebook Body Tracker at a given point in time. The following are the parameters for this function call:

XrBodyTrackerFBbodyTracker: A handle to a body tracker created on a successful call to xrCreateBodyTrackerFB.

const XrBodyJointsLocateInfoFB* locateInfo: A structure that contains the requested time and space for the required body tracking positions. Callers should request a time equal to the predicted display time for the rendered frame. The system will employ appropriate modeling to provide body tracking for this time.

XrBodyJointLocationsFB* locations: The XrBodyJointLocationsFB structure that contains an array of returned body joint poses and tracker status (refer to Data Structure section for more details). The elements in these arrays are ordered according to the XrBodyJointsFB enum.

Body Tracking Data Structures

typedef struct XrSystemBodyTrackingPropertiesFB{
   XrStructureType type;
   void* XR_MAY_ALIAS next;
   XrBool32 supportsBodyTracking;
} XrSystemBodyTrackingPropertiesFB;

XrSystemBodyTrackingPropertiesFB is a structure used to describe if a system supports body tracking. It is obtained through a call to xrGetSystemProperties It has the following members:

XrStructureType type: The XrStructureType of this structure.

void* XR_MAY_ALIAS next: NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

XrBool32 supportsBodyTracking: A boolean indicating if the body tracking is supported by the device/system and the application has declared the usage of this feature by requesting permission com.oculus.permission.BODY_TRACKING in the manifest. TRUE indicates support while FALSE indicates no support. A call to xrCreateBodyTrackerFB will fail with a failure XrResult of XR_ERROR_FEATURE_UNSUPPORTED if the system does not support body tracking.

typedef struct XrBodyTrackerCreateInfoFB{
   XrStructureType type;
   const void* XR_MAY_ALIAS next;
   XrBodyJointSetFB bodyJointSet;
} XrBodyTrackerCreateInfoFB;

XrBodyTrackerCreateInfoFB is a structure used to describe the requested functionality of the body tracker. It has the following members:

XrStructureType type: The XrStructureType of this structure.

const void* XR_MAY_ALIAS next: NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

XrBodyJointSetFB bodyJointSet: The requested set of joints for the tracker. See XrBodyJointSetFB for more information.

typedef struct XrBodyJointsLocateInfoFB{
   XrStructureType type;
   const void* XR_MAY_ALIAS next;
   XrSpace baseSpace;
   XrTime time;
} XrBodyJointsLocateInfoFB;

XrBodyJointsLocateInfoFB is a structure used to describe the time at which body joints are being requested. Callers should request a time equal to the predicted display time for the rendered frame. The system will employ appropriate modeling to provide body joints for this time. It has the following members:

XrStructureType type: The XrStructureType of this structure.

const void* XR_MAY_ALIAS next: NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

XrSpace baseSpace: The XrSpace within which the returned body joints poses will be represented.

XrTime time: The time at which the body joints are requested. Callers should request a time equal to the predicted display time for the rendered frame. The system will employ appropriate modeling to provide expressions for this time.

typedef struct XrBodyJointLocationFB{
    XrSpaceLocationFlags locationFlags;
    XrPosef pose;
} XrBodyJointLocationFB;

XrBodyJointLocationFB is a structure that contains the position, orientation and tracking status of a specific body joint. It has the following members:

XrSpaceLocationFlags locationFlags: Flags that indicate the tracking status and validity of the joint location (position, rotation).

XrPosef pose: An XrPosef pose describing the position and orientation of a body joint. The pose is represented in the coordinate system provided by baseSpace in the XrBodyJointsLocateInfoFB structure.

typedef struct XrBodyJointLocationsFB {
    XrStructureType type;
    void* XR_MAY_ALIAS next;
    XrBool32 isActive;
    float confidence;
    uint32_t jointCount;
    XrBodyJointLocationFB* jointLocations;
    uint32_t skeletonChangedCount;
    XrTime time;
} XrBodyJointLocationsFB;

XrBodyJointLocationsFB is a structure that contains arrays describing the user’s body joint locations, along with confidence and validity values of the output body pose. It has the following members:

XrStructureType type: The XrStructureType of this structure.

void* XR_MAY_ALIAS next: NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

XrBool32 isActive: A bool to indicate if the returned body pose is active (tracked). Callers should check the validity of the pose prior to use.

float confidence: A float between 0 and 1 which represents the confidence for the returned body pose. A value of 0 means no confidence in the pose returned, and a value of 1 means maximum confidence in the returned body pose.

uint32_t jointCount: An integer that represents the number of joints in jointLocations. If jointCount is not equal to XR_BODY_JOINT_COUNT_FB the system will return XR_ERROR_VALIDATION_FAILURE.

XrBodyJointLocationFB* jointLocations: An array of XrBodyJointLocationFB that contains the position and orientation of each body joint. The order has a direct correspondence to the enum defined in XrBodyJointsFB.

uint_32t skeletonChangeCount: A counter which is incremented whenever the tracking auto calibrates the user’s skeleton scale and proportions.

XrTime time: Upon return, the value will be set to the time at which the joints were tracked. This time may not be the same as the requested time if the system was unable to extrapolate joints to the requested time.

typedef enum XrBodyJointSetFB{
   XR_BODY_JOINT_SET_DEFAULT_FB = 0,
   XR_BODY_JOINT_SET_MAX_ENUM_FB = 0x7FFFFFFF
} XrBodyJointSetFB;

XrBodyJointSetFB is an enum used to describe the set of joints to be output by the tracker. It has the following possible values:

XR_BODY_JOINT_SET_DEFAULT_FB: The default set of joints.

XR_BODY_JOINT_SET_MAX_ENUM_FB: The maximum number of sets of joints. Currently, there is only one set supported (Default=0), which is described under XrBodyJointFB.

typedef struct XrBodySkeletonJointFB {
   int32_t joint;
   int32_t parentJoint;
   XrPosef pose;
} XrBodySkeletonJointFB;

The XrBodySkeletonFB structure is a container which represents the body skeleton in T-pose including the joint hierarchy.

int32_t joint: An index of a joint using the corresponding body joint enum described by XrBodyJointFB.

int32_t parentJoint: An index of a parent joint of the current joint using the corresponding body joint enum described by XrBodyJointFB.

XrPosef pose: A XrPosef defining the position and orientation of the origin of a body joint within the reference frame of the corresponding XrBodyJointsLocateInfoFB::baseSpace.

typedef struct XrBodySkeletonFB {
   XrStructureType type;
   void* XR_MAY_ALIAS next;
   uint32_t jointCount;
   XrBodySkeletonJointFB* joints;
} XrBodySkeletonFB;

The XrBodySkeletonFB structure is a container which represents the body skeleton in T-pose including the joint hierarchy.

XrStructureType type: The XrStructureType of this structure.

void* XR_MAY_ALIAS next: NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

uint32_t jointCount: An integer describing the count of elements in the joints array.

XrBodySkeletonJointFB joints: A pointer to an application allocated array of XrBodyskeletonJointFB that will be filled with skeleton joint elements upon successful completion of a call to xrGetSkeletonFB.

typedef enum XrBodyJointFB{
    XR_BODY_JOINT_ROOT_FB = 0,
    XR_BODY_JOINT_HIPS_FB = 1,
    XR_BODY_JOINT_SPINE_LOWER_FB = 2,
    XR_BODY_JOINT_SPINE_MIDDLE_FB = 3,
    XR_BODY_JOINT_SPINE_UPPER_FB = 4,
    XR_BODY_JOINT_CHEST_FB = 5,
    XR_BODY_JOINT_NECK_FB = 6,
    XR_BODY_JOINT_HEAD_FB = 7,
    XR_BODY_JOINT_LEFT_SHOULDER_FB = 8,
    XR_BODY_JOINT_LEFT_SCAPULA_FB = 9,
    XR_BODY_JOINT_LEFT_ARM_UPPER_FB = 10,
    XR_BODY_JOINT_LEFT_ARM_LOWER_FB = 11,
    XR_BODY_JOINT_LEFT_HAND_WRIST_TWIST_FB = 12,
    XR_BODY_JOINT_RIGHT_SHOULDER_FB = 13,
    XR_BODY_JOINT_RIGHT_SCAPULA_FB = 14,
    XR_BODY_JOINT_RIGHT_ARM_UPPER_FB = 15,
    XR_BODY_JOINT_RIGHT_ARM_LOWER_FB = 16,
    XR_BODY_JOINT_RIGHT_HAND_WRIST_TWIST_FB = 17,
    XR_BODY_JOINT_LEFT_HAND_PALM_FB = 18,
    XR_BODY_JOINT_LEFT_HAND_WRIST_FB = 19,
    XR_BODY_JOINT_LEFT_HAND_THUMB_METACARPAL_FB = 20,
    XR_BODY_JOINT_LEFT_HAND_THUMB_PROXIMAL_FB = 21,
    XR_BODY_JOINT_LEFT_HAND_THUMB_DISTAL_FB = 22,
    XR_BODY_JOINT_LEFT_HAND_THUMB_TIP_FB = 23,
    XR_BODY_JOINT_LEFT_HAND_INDEX_METACARPAL_FB = 24,
    XR_BODY_JOINT_LEFT_HAND_INDEX_PROXIMAL_FB = 25,
    XR_BODY_JOINT_LEFT_HAND_INDEX_INTERMEDIATE_FB = 26,
    XR_BODY_JOINT_LEFT_HAND_INDEX_DISTAL_FB = 27,
    XR_BODY_JOINT_LEFT_HAND_INDEX_TIP_FB = 28,
    XR_BODY_JOINT_LEFT_HAND_MIDDLE_METACARPAL_FB = 29,
    XR_BODY_JOINT_LEFT_HAND_MIDDLE_PROXIMAL_FB = 30,
    XR_BODY_JOINT_LEFT_HAND_MIDDLE_INTERMEDIATE_FB = 31,
    XR_BODY_JOINT_LEFT_HAND_MIDDLE_DISTAL_FB = 32,
    XR_BODY_JOINT_LEFT_HAND_MIDDLE_TIP_FB = 33,
    XR_BODY_JOINT_LEFT_HAND_RING_METACARPAL_FB = 34,
    XR_BODY_JOINT_LEFT_HAND_RING_PROXIMAL_FB = 35,
    XR_BODY_JOINT_LEFT_HAND_RING_INTERMEDIATE_FB = 36,
    XR_BODY_JOINT_LEFT_HAND_RING_DISTAL_FB = 37,
    XR_BODY_JOINT_LEFT_HAND_RING_TIP_FB = 38,
    XR_BODY_JOINT_LEFT_HAND_LITTLE_METACARPAL_FB = 39,
    XR_BODY_JOINT_LEFT_HAND_LITTLE_PROXIMAL_FB = 40,
    XR_BODY_JOINT_LEFT_HAND_LITTLE_INTERMEDIATE_FB = 41,
    XR_BODY_JOINT_LEFT_HAND_LITTLE_DISTAL_FB = 42,
    XR_BODY_JOINT_LEFT_HAND_LITTLE_TIP_FB = 43,
    XR_BODY_JOINT_RIGHT_HAND_PALM_FB = 44,
    XR_BODY_JOINT_RIGHT_HAND_WRIST_FB = 45,
    XR_BODY_JOINT_RIGHT_HAND_THUMB_METACARPAL_FB = 46,
    XR_BODY_JOINT_RIGHT_HAND_THUMB_PROXIMAL_FB = 47,
    XR_BODY_JOINT_RIGHT_HAND_THUMB_DISTAL_FB = 48,
    XR_BODY_JOINT_RIGHT_HAND_THUMB_TIP_FB = 49,
    XR_BODY_JOINT_RIGHT_HAND_INDEX_METACARPAL_FB = 50,
    XR_BODY_JOINT_RIGHT_HAND_INDEX_PROXIMAL_FB = 51,
    XR_BODY_JOINT_RIGHT_HAND_INDEX_INTERMEDIATE_FB = 52,
    XR_BODY_JOINT_RIGHT_HAND_INDEX_DISTAL_FB = 53,
    XR_BODY_JOINT_RIGHT_HAND_INDEX_TIP_FB = 54,
    XR_BODY_JOINT_RIGHT_HAND_MIDDLE_METACARPAL_FB = 55,
    XR_BODY_JOINT_RIGHT_HAND_MIDDLE_PROXIMAL_FB = 56,
    XR_BODY_JOINT_RIGHT_HAND_MIDDLE_INTERMEDIATE_FB = 57,
    XR_BODY_JOINT_RIGHT_HAND_MIDDLE_DISTAL_FB = 58,
    XR_BODY_JOINT_RIGHT_HAND_MIDDLE_TIP_FB = 59,
    XR_BODY_JOINT_RIGHT_HAND_RING_METACARPAL_FB = 60,
    XR_BODY_JOINT_RIGHT_HAND_RING_PROXIMAL_FB = 61,
    XR_BODY_JOINT_RIGHT_HAND_RING_INTERMEDIATE_FB = 62,
    XR_BODY_JOINT_RIGHT_HAND_RING_DISTAL_FB = 63,
    XR_BODY_JOINT_RIGHT_HAND_RING_TIP_FB = 64,
    XR_BODY_JOINT_RIGHT_HAND_LITTLE_METACARPAL_FB = 65,
    XR_BODY_JOINT_RIGHT_HAND_LITTLE_PROXIMAL_FB = 66,
    XR_BODY_JOINT_RIGHT_HAND_LITTLE_INTERMEDIATE_FB = 67,
    XR_BODY_JOINT_RIGHT_HAND_LITTLE_DISTAL_FB = 68,
    XR_BODY_JOINT_RIGHT_HAND_LITTLE_TIP_FB = 69,
    XR_BODY_JOINT_COUNT_FB = 70,
    XR_BODY_JOINT_NONE_FB = -1,
    XR_BODY_JOINT_MAX_ENUM_FB = 0x7FFFFFFF
} XrBodyJointFB;

The structure that describes the 70 body joints supported by the body tracker (18 core body joints + 52 hand joints) .

Android Permissions

For applications which are running on the device, APIs that provide access to sensitive data are protected by Android permissions. This does not apply to applications running on a separate device and connecting to the headset using Link.

Face tracking data is protected by the permission com.oculus.permission.FACE_TRACKING
Eye tracking data is protected by com.oculus.permission.EYE_TRACKING

Due to the sensitive nature of these data streams, these permissions’ protection level is set to “dangerous”. Read Android documentation to learn more about dangerous persmissions.

In order to access these APIs, your application must:

Declare the corresponding permission(s) in its manifestand
Request the permission(s) at runtime

Declaring Permissions

To declare a permission, add a <uses-permission> entry to the application’s Android manifest. This is typically located in AndroidManifest.xml.

In this example, both eye and face tracking permissions are declared:

<manifest ... >
    <uses-permission android:name="com.oculus.permission.EYE_TRACKING"/>
    <uses-permission android:name="com.oculus.permission.FACE_TRACKING"/>
    ...
</manifest>

Requesting Permissions At Runtime

In order for a dangerous permission to be granted to your application, your application must also request the permission at runtime. To do so, the application will need to either call directly into the Android permissions API, or use an engine, library, or framework that in turn interfaces with the Android permissions API.

When the application requests a permission that it does not already have, the user will be prompted by a modal dialog asking to decide whether to grant the permission to the application. The application will then be notified of the user’s response. This notification is asynchronous when directly using the Android API.

The following example demonstrates how to request eye tracking permission in Java:

import android.content.pm.PackageManager;
import android.os.Bundle;

…

private static final String PERMISSION_EYE_TRACKING =
    "com.oculus.permission.EYE_TRACKING";
private static final int REQUEST_CODE_PERMISSION_EYE_TRACKING = 1;

…


  private void requestEyeTrackingPermissionIfNeeded() {
    if (checkSelfPermission(PERMISSION_EYE_TRACKING) !=
   PackageManager.PERMISSION_GRANTED) {
      requestPermissions(new String[] {PERMISSION_EYE_TRACKING},
       REQUEST_CODE_PERMISSION_EYE_TRACKING);
    }
  }

…

@Override
public void onRequestPermissionsResult(int requestCode, String[] permissions,
  int[] grantResults) {
    switch (requestCode) {
  case REQUEST_CODE_PERMISSION_EYE_TRACKING:
      if (grantResults.length > 0 &&
        grantResults[0] == PackageManager.PERMISSION_GRANTED) {
    // Permission is granted
      }
      …
  }
  …
    }
}

See the Android permissions documentation for more information on:

How to check whether the application already has a given permission (checkSelfPermission())
How to request one or more permissions (requestPermissions())
How to handle the user’s response to the application’s permission request(s) (onRequestPermissionsResult())

Note that these APIs may vary slightly depending on the Android API version your application is targeting. VROS is currently based on Android 10, which is API level 29.

Use the Android permissions documentation to learn more about best practices on requesting permissions at runtime.

Did you find this page helpful?