Using passthrough, users can see their physical surroundings through a video feed from the Meta Quest’s cameras. These mixed reality experiences blend digital objects with physical environments.

Passthrough is not a simple video feed displayed to the user. To deliver realistic, comfortable, and natural viewing experiences, it is important that passthrough provides realistic depth and proper distortion. Simple monoscopic video (i.e., seen from a single viewing angle) fails to convey a sense of depth, and can be uncomfortable or unsafe for users.

Passthrough is a crucial element of mixed reality because it provides a video feed of the physical environment that can be combined with digital objects. Being able to use the physical environment around the user opens up exciting new possibilities for designing content and interactions. For example, a digital recipe can be overlaid on a physical surface in a kitchen, or a movie can be watched while a user is interacting with their family.

When the passthrough feed is used as an entire background, users can see their 360 degree surroundings in the physical environment, just like wearing see-through glasses. By adding virtual content on top of this passthrough background, it’s possible to build mixed reality experiences.

Passthrough helps build apps that improve physical environment awareness while enjoying virtual experiences. For example, the productivity application below shows virtual office tools within the user’s actual, physical workspace.

The selective passthrough capability uses a passthrough feed on a specific portion of a fully immersive experience. For example, magical windows can be overlaid onto the physical world while in fully immersive experiences. This can help show only partial windows into the user’s physical environment, and also helps frame transitions between physical and fully immersive experiences. The following image shows clear separation along natural borders, or full screen blends where virtual and physical spaces align - revealing delightful experiences in a comfortable manner. Another way of only making passthrough visible within the specified surface geometries, is to use surface-projected passthrough which allows apps to specify the geometry onto which the passthrough images are projected.

When passthrough is used with scene understanding, virtual objects can be shown interacting with physical surfaces. For example, a character can walk on any surface, including walls and tables. By combining passthrough and scene understanding, the physical environment can be used as a design canvas.

If an app requires frequent switching between the passthrough and fully immersive experiences, consider providing an easily accessible toggle-switch UI, such as a main menu or manual menu. The abrupt transition between virtual and physical environments can be an overwhelming and uncomfortable experience, especially when the two environments look vastly different. Because of this, it’s a good idea to apply smooth blending when transitioning between the two.

Instead of completely turning the passthrough feed on-and-off, you can also control the opacity of the passthrough feed. This means, you can blend the physical and virtual experiences smoothly. As described in the section above, be mindful that overlapping two distinct and incongruent realities can be confusing for users. Controlling the opacity of passthrough can be useful when the virtual environment aligns 1:1 with the user’s physical environment.

If passthrough is not a critical part of your app, and does not require frequent switching, the toggle option can be provided in the settings menu.

For mixed reality apps that launch directly into a passthrough view, the user experience is typically much better if the loading screen shows passthrough in the background. Before the user loads the app, the Meta Horizon OS can detect if the user is in passthrough mode in their home environment. If so, the splash screen that appears when loading the app can be displayed in passthrough as well. This is called contextual passthrough.

When presenting users with multiple realities, a good rule of thumb is to separate them and clearly mark the transition. The exception to the above rule is when a 1:1 match exists between virtual and physical spaces. In these cases, the effect can be magical. Generally speaking, overlapping two distinct realities (the physical and virtual worlds) can be confusing for users, and is discouraged. The The World Beyond sample app demonstrates a good example of delightful transition between the passthrough physical world and virtual world with animation and particle effects.

Be mindful that overlapping two distinct and incongruent realities (the physical and virtual worlds) can be confusing for users, and is discouraged. The following examples illustrate how the failure to separate and clearly mark the transition from virtual to physical spaces leads to a jarring, disorienting experience. This can cause users to feel confused and unsafe. In both images, a virtual grid overlays the physical world, making it difficult for a user to understand which reality to follow.

The following image show clear separation along natural borders, or full screen blends where virtual and physical spaces align - revealing delightful experiences in a comfortable manner.

The World Beyond sample app demonstrates a good example of delightful transition between the passthrough physical world and virtual world with animation and particle effects.

The Boundary API allows us to turn off the boundaries contextually. This means it turns on the boundaries when in fully immersive experiences and disabled the boundaries when the user sees their environment in passthrough.

Depending on the Meta Quest type, passthrough may be displayed in color or grayscale. The images below show examples of Meta Quest 2’s grayscale pass-through (left) and Meta Quest Pro’s color passthrough (right).

Passthrough can be shown as-is or stylized to achieve an artistic look. If using Meta Quest 2 with grayscale passthrough, certain considerations need to be kept in mind:

In the following image, a portal is about to open in the middle of the room. Color is used to draw the users attention.

On Meta Quest Pro, you can leverage a color passthrough video feed of the environment. It becomes easier to blend physical and virtual worlds together with this added level of detail.

Our mixed reality capabilities allow developers to control the blending between passthrough and their app. By playing with the transparency of a black background, applications can create the illusion that the environment is actually being lit and dimmed.

The following examples demonstrate lighting with passthrough. In the left-hand image, passthrough is dimmed everywhere, except for a cone of light attached to a flashlight that the user holds in their hand. In the right-hand image, passthrough is dimmed around a board game on a table, drawing the user’s attention to the game.

The World Beyond shows a dimmed environment, and encourages the user to focus on the portal where multi toy shows up. When the user grabs the multi toy, passthrough gradually adjusts the lighting with animation, creating a comfortable, intuitive experience.

Design seamless transitions between experiences by using gentle, predictable animations that help users visually track changes. Allow users to control their level of immersion by providing clear entry and exit points, such as prominent buttons or controls. Ensure that exiting an immersive experience is intuitive and doesn’t result in unexpected consequences, like quitting the app or game without saving progress. Consider allowing the user to customize the immersion range for an experience to enhance the user experience.