DISCLAIMER: The features described in this document are being released as an experimental feature. This experimental API allows you to start building and testing experiences using the Passthrough Camera API. But Meta Store will not accept any products that incorporate this experimental feature. This feature is provided on an “as is” basis, subject to all applicable terms set forth in the Oculus SDK License Agreement. While functional, this API may not reflect the quality or performance anticipated for the upcoming release, nor will it reflect typical behavior for system access (see note, below). By participating in this evaluation, you have the opportunity to provide feedback to influence our implementation, but acknowledge that the API may change before the final release.
NOTE: Passthrough Camera app access will not appear in settings. Once camera access is granted, the app with access must be uninstalled to revoke access.
Purpose and Scope
Building on top of the Android Camera2 API this release introduces the new feature of Passthrough Camera API, which provides access to the forward-facing cameras on the Quest 3 and Quest 3S for the purpose of supporting Computer Vision and Machine Learning. This API is distinct from the Media Projection API which supports casting from the User’s POV and should be used if your purpose is to represent what the user is seeing including the User Interface. We anticipate developers using the Passthrough Camera API to add application-specific computer-vision capabilities to extend the understanding of the user’s environment and actions beyond what is provided by Quest Scene API.
Use Cases
This API provides an unobstructed view using the forward-facing RGB cameras and can be integrated with ML/CV pipelines. Some common use cases that we anticipate are:
Specially-trained ML/CV models that can identify specific objects with which the application can interact (e.g. fitness equipment like dumbbells, industrial equipment like audio mixers).
ML/CV Assistant or Guides for experiences. For instance, a museum tour that when the user looks at a painting, they could get all kinds of information about the history of the painting, artist, style, using existing off-device LLMs.
Feedback for training or special-interest applications. After telling the user to do some task the app could detect if the task was done correctly by interpreting changes to the environment (e.g., writing on a whiteboard, changes to the objects in the room).
Design Improvements. By interpreting the lighting or using the camera image to modify a texture, much more realistic design effects can be achieved.
General Prerequisites
Horizon OS v74 or later
Quest 3 or Quest 3S
A new permission for headset cameras will be required: horizonos.permission.HEADSET_CAMERA
Passthrough Camera Using Android Camera2
The basis of Camera Access on Quest is our implementation of Android’s Camera 2 API within Horizon OS. Camera2 is implemented in most modern Android phones and there are many Open Source and commercial products that utilize Camera2. And starting from Horizon OS v74, Camera2 is also available on Quest headsets. We also added two special Meta Vendor Tags to distinguish passthrough cameras from other virtual or physical cameras on the headset.
The Android Camera2 API provides a set of features for building camera apps, including:
Image capture: capture camera data for advanced processing
Camera metadata: query the API for various hardware capabilities and configuration information
Multi-camera support: access and control multiple cameras
These capabilities enable developers to build feature-rich camera apps with fine-grained control over camera settings. On Quest 3 and Quest 3s developers have access to the left and right cameras on the face of the HMD.
Camera Vendor Tags
To help using the Android camera API, several “vendor tags” - Meta camera device specific tags - have been defined. The tag names can be accessed from the device’s CameraCharacteristics, using the name (Android SDK only) or descriptor (Android NDK). For further details please consult Android’s camera2 API documentation.
Name
Descriptor
Description
com.meta.extra_metadata.camera_source
0x80004d00
For Passthrough RGB camera this will always have value ‘0’
com.meta.extra_metadata.position
0x80004d01
For leftmost camera this has value ‘0’, for rightmost camera this has value ‘1’ from perspective of someone wearing the headset
Known Issues
After the first install of an app using the Camera API in v74 after flashing a new OS, the RGB channels might flip causing color distortion (e.g. flesh tones will appear bluish). This can be solved by rebooting the device.
Currently the Android Camera API is not supported over Link.
The passthrough camera texture captures a rectangular area (1280x960) smaller than what a user sees in the Quest 3. The top and bottom areas get cut off. We are investigating adding a square resolution (e.g. 1280x1280), but it is not provided in this release.
There is an order-of-operations that may cause the camera service to crash during startup. This crash will be auto-recovered and should not impact your app, but is noted here in case it is noticed in the debug logs.
Best Practices
Ensure user privacy: camera image data is considered Device User Data and is included in our Developer Data Use Policy. Adhere to this policy fully during your experimentation with the Camera Access on Quest.
Avoid costly processing on device: on-device processing of camera images can enable compelling use cases, but ensure that your experiences remain comfortable for users by maintaining a high framerate.
