Using the Dynamic Object Tracker
Updated: Jan 23, 2025
Dynamic Object Tracking enables your device to begin seeing real-world objects as they move in realtime, other than your hands or controllers.
Users may control whether or not the tracker is enabled and what objects they wish to allow applications to have access to. No tracking data is made available to applications without being enabled in the system settings.
At this time, the Meta Quest Dynamic Object Tracker supports the following tracked objects:
- Keyboards - includes laptop keyboards, bluetooth keyboards, and wired keyboards. May not detect uncommon keyboard types such as splitboards, unusual color schemes, or keyboards being used on a surface with low visual contrast.
This list may change in future updates as your device becomes even more useful.
Before you begin working with the Dynamic Object Tracker you will need the following:
Dynamic Object Tracker Prerequisites - A Meta Quest 3 or Meta Quest 3S running the latest version of the Meta Quest Operating System, v72 or newer
- Object tracking enabled in the System Settings by the user
- The application should enable the
XR_META_dynamic_object_tracker
extension and any object extensions desired, such as XR_META_dynamic_object_keyboard
- The Android manifest file for the application should mention both the
USE_ANCHOR_API
and USE_SCENE
permissions, to use the Dynamic Obect Tracker and access data through Scene apis.
<uses-permission android:name="com.oculus.permission.USE_ANCHOR_API" />
<uses-permission android:name="com.oculus.permission.USE_SCENE" />
- For Developers Only, the logged in Meta account must be registered with a developer organization. This will allow untrusted applications to use spatial data, which is required for this API. Applications downloaded from the Store or App Lab do not have this restriction.
To enable tracking for supported devices, do the following:
- In the headset, go to Settings > Devices > Keyboard
- Toggle the slider on this menu to enable Keyboard Tracking. Note that only keyboards are supported at this time, but this menu may change if more objects are available in future releases.
If a physical keyboard is visible in front of you, an outline should now appear in your home environment. With hand tracking enabled, move your hands toward the keyboard to show a passthrough cutout.
Using the Dynamic Object Sample App
With the Dynamic Object Tracker enabled, launch the XrDynamicObjects sample app which is included in the
Meta OpenXR Mobile SDK.
To use the tracker in applications that are not listed on the Store or App Lab, you must have a developer organization linked to your Meta account.
The XR_META_dynamic_object_tracker
 OpenXR extension makes available the API that applications may use to interact with the Dynamic Object Tracker. Please read the Prerequisites and Enable Tracking sections before attempting to use this API on your own device.
To see if the device supports Dynamic Object Tracking, the application should call xrGetSystemProperties
with an XrSystemProperties
struct correctly typed, and extended with a XrSystemDynamicObjectTrackerPropertiesMETA
also correctly typed.
The following functions are available with this extension enabled on supported devices:
XrResult xrCreateDynamicObjectTrackerMETA(
XrSession session,
const XrDynamicObjectTrackerCreateInfoMETA* createInfo,
XrDynamicObjectTrackerMETA* dynamicObjectTracker);
This function will immediately return a handle via the XrDynamicObjectTrackerMETA*
output parameter, and will asynchronously allocate resources needed for the tracker. The XrDynamicObjectTrackerCreateInfoMETA*
is provided for future extension, but currently does not offer any additional configuration. This function gives the system a hint that your application will use the tracker and reserves any needed resources, but does not actually begin tracking until the tracker has been configured.
typedef struct XrEventDataDynamicObjectTrackerCreateResultMETA {
XrStructureType type;
const void* XR_MAY_ALIAS next;
XrDynamicObjectTrackerMETA handle;
XrResult result;
} XrEventDataDynamicObjectTrackerCreateResultMETA;
The result of this function does not indicate a successful tracker bringup; the application should listen for an event of type XR_TYPE_EVENT_DATA_DYNAMIC_OBJECT_TRACKER_CREATE_RESULT_META
to confirm whether bringup was successful. If the XrResult
in this event is XR_ERROR_INITIALIZATION_FAILED
, the tracker has been disabled by the user in the system settings. This state may change throughout the lifetime of an application, so it may make sense for some apps to reattempt tracker creation periodically, like when switching menu screens or when a user toggles the app settings. Similarly, the user may disable the tracker mid-session. When this occurs, no new data will flow through the API, and existing object data will be cleared from memory; the application will not be made aware of this state unless the tracker is destroyed and xrCreateDynamicObjectTrackerMETA
is called again.
It is important to verify the handle in the event matches the handle created when the function was called, as you may recieve different handle values for each call to this function. Note that only one instance of the Dynamic Object Tracker is in use at any time, regardless of how many objects are being tracked.
Configuring Tracked Objects XrResult xrSetDynamicObjectTrackedClassesMETA(
XrDynamicObjectTrackerMETA dynamicObjectTracker,
const XrDynamicObjectTrackedClassesSetInfoMETA* setInfo);
You can tell the Dynamic Object Tracker what objects you want to track using the XrDynamicObjectTrackedClassesSetInfoMETA*
struct. The xrSetDynamicObjectTrackedClassesMETA
function will return XR_ERROR_FEATURE_UNSUPPORTED
if any value is passed in that does not map to a supported object type for the current headset. You can check whether an object class is supported by reviewing the enabling extension’s system properties.
typedef struct XrEventDataDynamicObjectSetTrackedClassesResultMETA {
XrStructureType type;
const void* XR_MAY_ALIAS next;
XrDynamicObjectTrackerMETA handle;
XrResult result;
} XrEventDataDynamicObjectSetTrackedClassesResultMETA;
Similar to the xrCreateDynamicObjectTrackerMETA
function, this function returns immediately, and a followup event will be sent with the result of the tracker configuration with the type XR_TYPE_EVENT_DATA_DYNAMIC_OBJECT_SET_TRACKED_CLASSES_RESULT_META
.
Querying for Tracked Objects You may query for tracked objects, review their components such as 2d or 3d bounding boxes, and obtain their 3d poses via the Scene API. If you are unfamiliar with Spatial Entities or Scene, you are strongly encouraged to review the
Scene Documentation before continuing. You may also reference the XrDynamicObject sample application included in the Mobile SDK for a hands-on learning approach. Dynamically tracked objects can be queried using the
xrQuerySpacesFB
function, but they do not support the related discover APIs at this time.
XrResult xrGetSpaceDynamicObjectDataMETA(
XrSpace space,
XrDynamicObjectDataMETA* dynamicObjectDataOutput);
The Scene API is extended by XR_META_dynamic_object_tracker
with a function to retrieve dynamically tracked object data, such as the object type.
XrResult xrDestroyDynamicObjectTrackerMETA(
XrDynamicObjectTrackerMETA dynamicObjectTracker);
Applications that are no longer using any dynamically tracked objects are encouraged to call xrDestroyDynamicObjectTrackerMETA
on their active tracker handle. This will free up any resources used by the tracker.
The XR_META_dynamic_object_keyboard
 OpenXR extension enables the device to track Keyboard types, including laptop keyboards, wireless keyboards, and wired keyboards. Only one keyboard is supported by the device at a time. While this tracker is generic, it may have trouble detecting unusual keyboards such as split keyboards, keyboards with unusual color schemes, or keyboards that are on a surface with low contrast, like a white keyboard on a white desk.
To see if the device supports Keyboard Tracking, the application should call xrGetSystemProperties
with an XrSystemProperties
struct correctly typed, and extended with a XrSystemDynamicObjectKeyboardPropertiesMETA
also correctly typed.
The enum XrDynamicObjectClassMETA
from XR_META_dynamic_object_tracker is extended with value XR_DYNAMIC_OBJECT_CLASS_KEYBOARD_META
.