Meta Developers

Eye Tracking for Meta Quest Pro detects eye movement and enables the Eye Gaze API to drive the eye transforms for a user’s embodied character as they look around. The abstracted eye gaze representation that the API provides allows a user’s character representation to make eye contact with other users, significantly improving social presence. It can also be used to tell where the person is looking in the 3D space which can provide a good understanding of regions of interest or even be used for targeting in games.

Policies and Disclaimers

Your use of the Eye Tracking API must at all times be consistent with the Oculus SDK License Agreement and the Developer Data Use Policy and all applicable Oculus and Meta policies, terms and conditions. Applicable privacy and data protection laws may cover your use of Movement, and all applicable privacy and data protection laws.

In particular, you must post and abide by a publicly available and easily accessible privacy policy that clearly explains your collection, use, retention, and processing of data through the Eye Tracking API. You must ensure that a user is provided with clear and comprehensive information about, and consents to, your access to and use of abstracted gaze data prior to collection, including as required by applicable privacy and data protection laws.

Please note that we reserve the right to monitor your use of the Eye Tracking API to enforce compliance with our policies.

When a user enables eye tracking for your app, your app is granted access to real time abstracted gaze data, which is user data under the Developer Data Use Policy. You are expressly forbidden from using this data for Data Use Prohibited Practices in accordance with the Developer Data Use Policy. The eye tracking feature is powered by our Eye Tracking API technology.

XrEyes Sample App

Build XrEyes Sample App

Download the Oculus Mobile OpenXR SDK (v47 or later) and then build the XrEyes sample application with:

adb uninstall com.oculus.sdk.xreyes
cd XrSamples/XrEyes/Projects/Android
../../../../gradlew installDebug

Using XrEyes Sample App

As a user, when you open the sample app, you will see a world-locked table with entries of eye gaze data for both eyes. For each eye, the rotation is listed in both 4 quaternions and 3 Euler angles along with the eye positions and the confidence value. The eye gaze in the app is specified in the world coordinate system.

As you move your head and track the world-locked table, you will notice that the eye gaze rotations barely change. This is expected and indicates that your eyes are being tracked correctly. Instead, if you move your eyes around and take a recording of your screen, then you will notice that the gaze is correctly updated. You will also see that the confidence value of the eye tracking is 0.5 and does not change. Again, this is expected, as the eye tracking confidence output by the tracker is not fully reliable currently and has been set to 0.5.

Permissions

For using eye tracking functionality in your app, you must declare the "com.oculus.permission.EYE_TRACKING" permission in the Android manifest.

<manifest xmlns:android="http://schemas.android.com/apk/res/android">
    <uses-feature android:name="oculus.software.eye_tracking" android:required="true" />
    <uses-permission android:name="com.oculus.permission.EYE_TRACKING" />

    ....
</manifest>

For details, read the OpenXR Support for Meta Quest Headsets guide.

The com.oculus.permission.EYE_TRACKING permission is a “runtime” permission, so the app must explicitly ask the user to grant permission. For details about runtime permissions, read Runtime permissions. Here is a sample Main Activity to handle permissions:

  private static final String PERMISSION_EYE_TRACKING = "com.oculus.permission.EYE_TRACKING";
  private static final int REQUEST_CODE_PERMISSION_EYE_TRACKING = 1;

  @Override
  protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    requestEyeTrackingPermissionIfNeeded();
  }

  private void requestEyeTrackingPermissionIfNeeded() {
    if (checkSelfPermission(PERMISSION_EYE_TRACKING) != PackageManager.PERMISSION_GRANTED) {
      requestPermissions(
          new String[] {PERMISSION_EYE_TRACKING}, REQUEST_CODE_PERMISSION_EYE_TRACKING);
    }
  }

The Eye Tracking Extension

XR_FB_eye_tracking_social introduces an extension to enable social applications distinct from the existing XR_EXT_eye_gaze_interaction extension which is meant for enabling interaction with eye gaze. The XR_FB_eye_tracking_social extension provides you with the gaze direction of each of the eyes along with their respective positions. In addition, the confidence value of each of the eye gaze is also output as a float with value in range [0, 1], although, currently, it is fixed to 0.5. Because social applications rely on fixating the gaze on a point/person in space, the gaze output from both eyes is temporally smoothed and necessarily converges in front of the user.

Because eye tracking relies on dedicated sensors on the device, the API is supported on Meta Quest Pro, but is not supported on Meta Quest 2 and older devices which do not have those sensors.

For full API reference, see Movement API reference.

Setting Up

Include Headers

In your source code, include the following header for eye tracking:

   #include <openxr/openxr.h>

Initialize OpenXR

Prior to using eye tracking, you must initialize an OpenXR session and enable the extension. For details about session initialization, read Creating Instances and Sessions.

Initialization

Before the app gets access to functions of a specific OpenXR extension, you must create the OpenXR session and enable the required OpenXR extension. That part of the application is common for all extensions.

During initialization, you can create the following set of objects, which will be shared between all OpenXR extensions of the app:

XrInstance instance;
XrSystemId system;
XrSession session;
XrSpace sceneSpace;

For details, see the SampleXrFramework/Src/XrApp.h header in the Oculus OpenXR Mobile SDK.

Process is described in OpenXR specification:

For implementation details, see SampleXrFramework/Src/XrApp.cpp in the Oculus OpenXR Mobile SDK.

Enable Extensions

All extensions should be explicitly listed for creating an XrInstance:

std::vector<const char*> extensions;

// ...

XrInstance Instance = XR_NULL_HANDLE;

XrInstanceCreateInfo instanceCreateInfo = {XR_TYPE_INSTANCE_CREATE_INFO};

// ...
instanceCreateInfo.enabledExtensionCount = extensions.size();
instanceCreateInfo.enabledExtensionNames = extensions.data();

// ...
OXR(initResult = xrCreateInstance(&instanceCreateInfo, &Instance));

For implementation details, see SampleXrFramework/Src/XrApp.cpp in the Oculus OpenXR Mobile SDK.

Eye Tracking Initialization

You must initialize the OpenXR extension once and share it between all calls to the OpenXR API. If you do it successfully, you will have the following data:

    XrSession Session;
    XrSpace StageSpace;

For details, see the SampleXrFramework\Src\XrApp.h header.

It is recommended that you use the constant XR_FB_EYE_TRACKING_SOCIAL_EXTENSION_NAME as an extension name.

Check Compatibility

You must check if the user’s headset supports eye tracking. For a given XrInstance, you must receive the system properties through calling the xrGetSystemProperties function to validate this.

To do so, use the XrSystemEyeTrackingPropertiesFB struct that describes if a system supports eye tracking. Its definition follows.

typedef struct XrSystemEyeTrackingPropertiesFB {
   XrStructureType type;
   void* XR_MAY_ALIAS next;
   XrBool32 supportsEyeTracking;
} XrSystemEyeTrackingPropertiesFB;

For details about this struct, see XrSystemEyeTrackingPropertiesFB in Movement API reference.

The following example demonstrates how to validate eye tracking support.

        XrSystemEyeTrackingPropertiesFB eyeTrackingSystemProperties{
            XR_TYPE_SYSTEM_EYE_TRACKING_PROPERTIES_FB};
        XrSystemProperties systemProperties{
            XR_TYPE_SYSTEM_PROPERTIES, &eyeTrackingSystemProperties};
        OXR(xrGetSystemProperties(GetInstance(), GetSystemId(), &systemProperties));
        if (!eyeTrackingSystemProperties.supportsEyeTracking) {
            return;
        }

If the eyeTrackingSystemProperties field of XrSystemEyeTrackingPropertiesFB returns true, eye tracking is supported.

Acquire Function Pointers

To create the eye tracker, you must retrieve links to all the functions in the extension before usage. For details, read xrGetInstanceProcAddr in the OpenXR spec. Here is an example on how to achieve this.

    PFN_xrCreateEyeTrackerFB xrCreateEyeTrackerFB_ = nullptr;
    PFN_xrDestroyEyeTrackerFB xrDestroyEyeTrackerFB_ = nullptr;
    PFN_xrGetEyeGazesFB xrGetEyeGazesFB_ = nullptr;

        OXR(xrGetInstanceProcAddr(
            GetInstance(), "xrCreateEyeTrackerFB", (PFN_xrVoidFunction*)(&xrCreateEyeTrackerFB_)));
        OXR(xrGetInstanceProcAddr(
            GetInstance(),
            "xrDestroyEyeTrackerFB",
            (PFN_xrVoidFunction*)(&xrDestroyEyeTrackerFB_)));
        OXR(xrGetInstanceProcAddr(
            GetInstance(), "xrGetEyeGazesFB", (PFN_xrVoidFunction*)(&xrGetEyeGazesFB_)));

Using the Extension

Create Eye Tracker

You can create an eye tracker by calling the xrCreateEyeTrackerFB function. This function is used to create and obtain an XrEyeTrackerFB handle to an eye tracker. The function definition follows.

XrResult xrCreateEyeTrackerFB(
	XrSession session,
	const XrEyeTrackerCreateInfoFB* createInfo,
	XrEyeTrackerFB* eyeTracker);

To call this function you must use an XrEyeTrackerCreateInfoFB struct which describes the requested capabilities in order to create the eye tracker. The struct’s definition follows:

typedef struct XrEyeTrackerCreateInfoFB {
   XrStructureType type;
   const void* XR_MAY_ALIAS next;
} XrEyeTrackerCreateInfoFB;

For details, read XrEyeTrackerCreateInfoFB in the Movement API reference. The following example demonstrates how to use these.

   XrEyeTrackerFB eyeTracker_ = XR_NULL_HANDLE;
         XrEyeTrackerCreateInfoFB createInfo{XR_TYPE_EYE_TRACKER_CREATE_INFO_FB};
         OXR(xrCreateEyeTrackerFB_(GetSession(), &createInfo, &eyeTracker_));

Only one eye tracker per session is allowed and multiple calls to the xrCreateEyeTrackerFB function will return the same handle. The handle is unique per process and cannot be shared across processes.

Important: In order for xrCreateEyeTrackerFB function calls to succeed, apps must request the com.oculus.permission.EYE_TRACKING permission in their manifest, and a user must grant this permission.

Retrieve Eye Gaze Data

The eye gaze data, including the position of the eye, is computed and provided by the eye tracker for each eye. The eye gaze data from eye tracking will be available immediately through the xrGetGazesFB function call upon completion of the call to xrCreateEyeTrackerFB. Calls to the xrGetEyeGazeFB function obtain pose for the eyes of a user at a specific time and within a specific coordinate system.

The definition of xrGetGazesFB function is the following:

XrResult xrGetEyeGazesFB(
   XrEyeTrackerFB eyeTracker,
   const XrEyeGazesInfoFB* gazeInfo,
   XrEyeGazesFB* eyeGazes);

For details, see xrGetGazesFB in Movement API reference. The XrEyeGazesInfoFB struct, which is a parameter toxrGetGazesFB, contains the requested time and space for the required eye gaze positions. Callers should request a time equal to the predicted display time for the rendered frame. The system will then employ appropriate modeling to provide eye gaze for this time. The definition of the XrEyeGazesInfoFB struct follows:

typedef struct XrEyeGazesInfoFB {
   XrStructureType type;
   const void* XR_MAY_ALIAS next;
   XrSpace baseSpace;
   XrTime time;
} XrEyeGazesInfoFB;

Calling the xrGetGazesFB function returns an XrEyeGazesFB struct which contains an array of gaze pose and confidence per eye and return timestamps. The XrEyeGazesFB struct is defined as:

typedef struct XrEyeGazesFB {
   XrStructureType type;
   void* XR_MAY_ALIAS next;
   XrEyeGazeFB gaze[XR_EYE_INDEX_COUNT_FB];
   XrTime time;
} XrEyeGazesFB;

For details about this struct, read XrEyeGazesFB in Movement API reference. Note: Index 0 of the gaze array represents the user’s left eye and index 1 represents the right eye.

The following example demonstrates how to call the xrGetEyeGazesFB function.

XrEyeGazesFB eyeGazes{XR_TYPE_EYE_GAZES_FB};
eyeGazes.next = nullptr;

XrEyeGazesInfoFB gazesInfo{XR_TYPE_EYE_GAZES_INFO_FB};
gazesInfo.baseSpace = GetStageSpace();

OXR(xrGetEyeGazesFB_(eyeTracker_, &gazesInfo, &eyeGazes));

For using eye tracking data, you must ensure that the gaze is valid. The general pattern is the following:

for (int eye = 0; eye < 2; ++eye) {
    if (eyeGazes.gaze[eye].isValid) {
        // eyeGazes.gaze[eye].gazePose.orientation contains orientation of an eye gaze
        // eyeGazes.gaze[eye].gazeConfidence contains confidential
        ....
    }
}

Destroy tracker

It is recommended to release resources before finishing the application by using the xrDestroyEyeTrackerFB function.

OXR(xrDestroyEyeTrackerFB_(eyeTracker_));

Did you find this page helpful?