Getting Started

1. Download a sample project and set it up in Unity Editor to explore a passthrough camera using the PassthroughCameraAccess component.
2. Display the passthrough texture on a 2D canvas.
3. Get the exact pose of an RGB camera in the world-space coordinates and how to orient 2D camera images accurately relative to the physical environment.
4. Understand how to access camera data on CPU to write a simple brightness estimation logic.
5. Use Unity Sentis to detect real-world objects with ML/CV.
6. Write a simple GPU shader to add effects to the passthrough camera.

1. Set up and gain access to the Passthrough Camera API using Unity.
2. Understand how to integrate with other Meta Quest APIs.
3. Understand the organization of the samples and the primary function of each of the samples.

• CameraViewer sample: shows a 2D canvas with the camera data inside.
• CameraToWorld sample: demonstrates how to align the pose of the RGB camera images with Passthrough, and how 2D image coordinates can be transformed into 3D rays in world space.
• BrightnessEstimation sample: illustrates brightness estimation and how it can be used to adapt the experience to the user’s environment.
• MultiObjectDetection sample: shows how to feed camera data to Unity Sentis to recognize real-world objects. For more information on the Sentis implementation in this sample, please refer to Unity Sentis page.
• ShaderSample: demonstrates how to apply custom effects to camera texture on GPU.

1. Configuring a project.
2. PassthroughCameraAccess component overview.
3. Using PassthroughCameraAccess methods to get camera extrinsics, intrinsics, and timestamps.
4. Overview of the samples that we have created to get you started.

• 320 x 240
• 640 x 480
• 800 x 600
• 1280 x 960

1. Clone the GitHub project: https://github.com/oculus-samples/Unity-PassthroughCameraApiSamples
2. Open the project with Unity 2022.3.58f1 or Unity 6000.0.38f1. Note: When creating a new Unity project, you must install the MRUK package to access the PassthroughCameraAccess component. See Mixed Reality Utility Kit - Getting Started for installation instructions.
3. Open ‘Meta / Tools / Project Setup Tool’ and fix any issues that it finds in the configuration of your project.
4. Create a new empty scene.
5. Select Meta > Tools > Building Blocks from the menu to add the Camera Rig and Passthrough building blocks to your scene. Camera Passthrough API depends on an application running with Passthrough enabled. See Configure your Unity project for instructions on enabling it.
6. To integrate Passthrough Camera API, select the ‘Camera Rig’ object in your scene, then enable ‘OVR Manager / Enabled Passthrough Camera Access’ setting and add the PassthroughCameraAccess component to a GameObject in your scene.
7. To access the camera texture from a custom C# script, get a reference to the PassthroughCameraAccess component and call its GetTexture() method. The method will return a valid texture only after “horizonos.permission.HEADSET_CAMERA” permission has been granted and the PassthroughCameraAccess is enabled. For example, in the CameraViewer example, we assign the texture to the RawImage.texture to display it with the Unity UI system.

• Initializing the camera to access the camera data through MRUK.
• Managing camera lifecycle when the component is enabled/disabled or the application is paused.
• Surfacing camera properties including intrinsics, extrinsics, timestamps, and real-time texture data.

• CameraPosition: camera source selection (Left or Right).
• RequestedResolution: desired resolution of the camera images. If the resolution is unsupported, the closest lower resolution is picked instead.
• MaxFramerate: maximum camera stream framerate. Default value is 60 FPS.
• TargetMaterial: automatic update of optional material with the camera texture.
• TexturePropertyName: texture property name to update. Default value is “_MainTex”.
• GetSupportedResolutions(CameraPositionType cameraPosition) - static method that returns an array of all supported resolutions.
• Intrinsics - property that returns the camera intrinsics data: FocalLength, PrincipalPoint, SensorResolution, and LensOffset.

• GetCameraPose() - returns the world pose of the passthrough camera at the current timestamp.
• ViewportPointToRay(Vector2 viewportPoint) - returns a 3D ray in world space which starts from the passthrough camera origin and passes through the viewport point.
• WorldToViewportPoint(Vector3 worldPosition) - transforms a world position to normalized viewport coordinates.
• Timestamp - property that provides the timestamp of the latest camera image.

// Unity, C#:
// Convert screen coordinates to normalized viewport coordinates
var viewportPoint = new Vector2((float)x / cameraAccess.CurrentResolution.x, (float)y / cameraAccess.CurrentResolution.y);
var ray = cameraAccess.ViewportPointToRay(viewportPoint);

if (environmentRaycastManager.Raycast(ray, out EnvironmentRaycastHit hitInfo))
{
    // Place a GameObject at the place of intersection
    anchorGo.transform.SetPositionAndRotation(
        hitInfo.point,
        Quaternion.LookRotation(hitInfo.normal, Vector3.up));
}

CameraViewer sample: a sample for getting the camera data and updating a Unity RawImage UI element.

While the sample is running, press A to pause or resume the camera feed. Press B to toggle the debug mode. While in the debug mode, all the scene objects are offset by 40 cm outwards, allowing you to see objects at the edges of the screen.

MultiObjectDetection sample shows how to feed the camera data to Unity Sentis to detect objects with ML/CV.

ShaderSample sample shows how to use the camera image on GPU as a regular UnityEngine.Texture inside a shader to create different kinds of effects.

1. Check the logs if you encounter errors or crashes. Both the sample and the PassthroughCameraAccess have lots of descriptive log messages that should be able to help you narrow down the problem.
2. Make sure ‘horizonos.permission.HEADSET_CAMERA’ Android permission is granted to your app. See the Managing Permissions section for instructions on how to manually grant permissions via the command line for debugging.
3. When updating the project to Unity version 6 or later, the Android manifest needs to be updated. This can be done either manually or by following these steps:
   1. Navigate to Meta > Tools > Update AndroidManifest.xml or Meta > Tools and click Create store-compatible AndroidManifest.xml.
   2. Add the ‘horizonos.permission.HEADSET_CAMERA’ permission back into the manifest manually after updating.
   Fix all warnings and errors in the Project Setup Tool after opening the project in the Unity version 6 or later.