Building on top of the Android Camera2 API this release introduces the new feature of Passthrough Camera API, which provides access to the forward-facing cameras on the Quest 3 and Quest 3S for the purpose of supporting Computer Vision and Machine Learning. This API is distinct from the Media Projection API which supports casting from the User’s POV and should be used if your purpose is to represent what the user is seeing including the User Interface. We anticipate developers using the Passthrough Camera API to add application-specific computer-vision capabilities to extend the understanding of the user’s environment and actions beyond what is provided by Quest Scene API.

Use Cases

This API provides an unobstructed view using the forward-facing RGB cameras and can be integrated with ML/CV pipelines. Some common use cases that we anticipate are:

Specially-trained ML/CV models that can identify specific objects with which the application can interact (e.g. fitness equipment like dumbbells, industrial equipment like audio mixers).

ML/CV Assistant or Guides for experiences. For instance, a museum tour that when the user looks at a painting, they could get all kinds of information about the history of the painting, artist, style, using existing off-device LLMs.

Feedback for training or special-interest applications. After telling the user to do some task the app could detect if the task was done correctly by interpreting changes to the environment (e.g., writing on a whiteboard, changes to the objects in the room).

Design Improvements. By interpreting the lighting or using the camera image to modify a texture, much more realistic design effects can be achieved.

General Prerequisites

Horizon OS v74 or later

Quest 3 or Quest 3S

A new permission for headset cameras will be required: horizonos.permission.HEADSET_CAMERA

Passthrough feature: must be enabled to access the Passthrough Camera API.

Passthrough Camera Using Android Camera2

The basis of Camera Access on Quest is our implementation of Android’s Camera 2 API within Horizon OS. Camera2 is implemented in most modern Android phones and there are many Open Source and commercial products that utilize Camera2. And starting from Horizon OS v74, Camera2 is also available on Quest headsets. We also added two special Meta Vendor Tags to distinguish passthrough cameras from other virtual or physical cameras on the headset.

The Android Camera2 API provides a set of features for building camera apps, including:

Image capture: capture camera data for advanced processing

Camera metadata: query the API for various hardware capabilities and configuration information

Multi-camera support: access and control multiple cameras

These capabilities enable developers to build feature-rich camera apps with fine-grained control over camera settings. On Quest 3 and Quest 3s developers have access to the left and right cameras on the face of the HMD.

Camera Vendor Tags

To help using the Android camera API, several “vendor tags” - Meta camera device specific tags - have been defined. The tag names can be accessed from the device’s CameraCharacteristics, using the name (Android SDK only) or descriptor (Android NDK). For further details please consult Android’s camera2 API documentation.

Name	Descriptor	Description
com.meta.extra_metadata.camera_source	0x80004d00	For Passthrough RGB camera this will always have value ‘0’
com.meta.extra_metadata.position	0x80004d01	For leftmost camera this has value ‘0’, for rightmost camera this has value ‘1’ from perspective of someone wearing the headset

Known Issues

After the first install of an app using the Camera API in v74 after flashing a new OS, the RGB channels might flip causing color distortion (e.g. flesh tones will appear bluish). This can be solved by rebooting the device.

Currently the Android Camera API is not supported over Link.

The passthrough camera texture captures a rectangular area (1280x960) smaller than what a user sees in the Quest 3. The top and bottom areas get cut off. We are investigating adding a square resolution (e.g. 1280x1280), but it is not provided in this release.

There is an order-of-operations that may cause the camera service to crash during startup. This crash will be auto-recovered and should not impact your app, but is noted here in case it is noticed in the debug logs.

Restrictions applied by the parent to restrict Teen and Youth accounts (e.g., accessing the Passthrough Camera API) are not applied if the application is installed through the Meta Quest Developer Hub application.

Closing an app that uses Camera Access, then quickly launching it again within a few seconds may cause a crash. This issue will be resolved in future releases.

Best Practices

Ensure user privacy: camera image data is considered Device User Data and is included in our Developer Data Use Policy. Adhere to this policy fully during your experimentation with the Camera Access on Quest.

Avoid costly processing on device: on-device processing of camera images can enable compelling use cases, but ensure that your experiences remain comfortable for users by maintaining a high framerate.