Meta Developers

Design

Scene

Scene allows users to interact with physical objects in immersive experiences.

With manual scene capture, users can define a roombox and bring in static objects like their couch, table, desk, etc.
Assisted scene capture (Meta Quest 3) captures a scene mesh and roombox information on the fly without users needing to manually define physical objects in their space.
The scene mesh supports interactions of virtual content with physical objects (e.g., motion rebound), but not proper occlusion.
Manual capture is required for proper occlusions between virtual content and individual physical objects in the activity space.
Marking dynamic objects (pets, robot vacuums, etc.) visible is not supported by either capture method; you should not create experiences that include dynamic physical objects without enabling Depth API.
Users should be encouraged to remove all objects from their activity space except those intended for interaction.
We do not recommend building experiences requiring higher precision activity beyond walking within the scene due to technical limitations.
Scene health and safety considerations depend on the type of experience: immersive mixed reality or passthrough primary mixed reality.

Immersive mixed reality experiences

In immersive mixed reality, the virtual environment may hide many (if not all) details of the physical world from view. For instance, when there is minimal passthrough, the scene is entirely re-skinned (e.g., covers the floor, walls and furniture) or is highly stylized. When re-skinning, it is important to closely align the virtual and physical environments; however, technological limitations do not allow for the Scene model to be a 1:1 match. Drift and misalignment can occur, causing the Scene model to become an inaccurate representation of the physical world over time.

Stylization is another form of immersive mixed reality that applies filters (e.g., color, tint, custom shaders) or custom shaders to passthrough to the display, but is also limited in its capabilities. We recommend a boundary at all times in immersive mixed reality for the following reasons:

User risk increases as more physical objects are left within the activity space.
The more physical objects left within the boundary that are not represented by virtual content, the greater the risk to users (e.g., the physical objects are not visible to the user and are “covered” or occluded by virtual content).
The degree of stylization of passthrough can also impact the visibility of the real world and a user’s ability to interact with physical objects.

Figure 1: An example of immersive mixed reality where custom shaders can be overlaid on the environment akin to re-skinning with scene. In both cases, object visibility and important navigational cues are reduced.

A living room with different passthrough tint, color, and edges

Figure 2: Basic stylized passthrough manipulates features such as tint, color, edges, etc

Assisted scene capture (Meta Quest 3)

Assisted Scene Capture (ASC) is a feature on Meta Quest 3 that automates the scene process by capturing scene mesh and roombox information on the fly, requiring the user to move around their space.

Do not use scene mesh for reskinning or stylizing the environment.
ASC is not designed for this function – it can lead to improper occlusion and difficulty navigating.
Avoid placing virtual content on or very close to walls (roombox) that users need to directly interact with from a close distance.
Content on the walls that users are encouraged to interact with increases the chance of controller, HMD and bodily contact.
Content that contains depth affordance (e.g., portals) may also entice users to peer through or walk into the wall (see Visual vs. physical affordance).
ASC is not a safety feature – it only defines physical objects to help make MR experiences more immersive. It does not provide safety warnings when users get close to physical objects.
Note that ASC should not be confused with smart boundary (SB) which is a safety feature. SB defines a playable space and provides cues to users about their proximity to physical objects in their playspace.

Passthrough primary mixed reality experiences

In some cases, passthrough is the primary layer of the visual display. Unlike immersive mixed reality, such experiences may be suitable to remove the boundary. Before doing so, however, it is critical to understand important safety considerations and make an informed decision on whether best practices are met.