Meta Developers

Design

Depth

Occlusion is a key component to enabling both believability and user safety in mixed reality. Depth API provides application developers real-time, environment depth estimates from the point of view of the headset.

Depth API supports dynamic occlusion, the occluding physical objects move (e.g., other people, pets, limbs) relative to the user; however, the depth subsystem has limited accuracy beyond four meters.
Depth API does not support moving virtual content bouncing off of physical objects; when used alone, virtual content will simply move through physical objects if they are closer in depth and make them invisible to the user.
Scene only supports static occlusion with scanned objects in the activity space but does support dynamic interactions between virtual content and the physical environment (i.e., rebounding).
Combining Depth API and Scene supports proper occlusion and increased immersion in mixed reality (see Occlusions with virtual content for important considerations).
Our Depth subsystem does not work well with transparent or reflective surfaces. Care should be taken to avoid placing virtual content near these objects/surface types.
Our Depth subsystem relies on machine learning to create a depth estimate. This sometimes leads to inaccurate estimates with small or thin physical objects such as pens and stair railings.

Occlusion with virtual content

Proper depth ordering

Proper depth ordering occurs when virtual content blocks (or hides) the view of the physical environment behind it or when virtual content is hidden by physical objects in front of it.

The occlusion feature in Depth API can prevent users from seeing and avoiding potential obstacles (e.g., walls, furniture, floor objects).
Immersion and current task demands can predominate the user’s mental model of the physical activity space.

Improper depth ordering

Improper depth ordering can occur when the HMD lacks knowledge of the environment.
Without occlusion using Depth API, virtual content can block physical objects that are in fact closer to the user; with Occlusion using Depth API, physical objects will hide or show virtual content based on their relative location in the depth map.
Scene information helps occlusions be more accurate if the object is marked and static, but it does not account for dynamic objects (e.g., people, pets).
Improper depth order may sometimes cause visual discomfort (e.g., unpleasant feeling of crossing your eyes).
Even with the Occlusion feature enabled, physical objects can still be immediately behind virtual ones; it is important to place virtual objects at a distance from physical ones, especially in experiences with high movement demands.

Figure 1: Improper depth order of a dynamic obstacle - a virtual panel blocks the user’s pet, though it is in fact farther away, preventing them from detecting the dog between them and the portal.

With Depth API enabled in passthrough, proper occlusion would be rendered and in turn, the pet would be visible and perceived as being in front of the panel.

A virtual character appears beyond the wall surface but blocks the segment of wall between it and the user

Figure 2: Improper occlusion of the wall – a virtual character appears beyond the wall surface but blocks the segment of wall between it and the user, giving the impression that there is actionable space between them and the virtual character. Virtual content should obey the physicality of the Scene model.

Figure 3: Improper occlusion of obstacles on the floor – both the dog and table are hidden by the virtual content. This creates a trip hazard.

Dynamic occlusion

Dynamic occlusion occurs when the occluding physical object is moving relative to the viewing user.

Different portions of activity space become visible/invisible at different times.
Though previously hidden physical objects can become visible, users may give precedence to virtual content and ignore the physical world.
Occlusion depends on the size, number, and distance of virtual objects relative to the user.
- For instance, one can completely cover their eyes with their hand, but that same hand blocks only a small portion of the FoV when held at arm’s length.
- Even if virtual content is limited to small objects, many of them together can block a high proportion of the physical environment.
- There is no one size fits all amount of acceptable occlusion (e.g., percentage of occupied pixels), but it should be considered in design based on user expectations in the experience.

Larger virtual content can hide the same percentage of FoV at a farther distance than smaller virtual content at a closer distance.

Figure 4: Larger virtual content can hide the same percentage of FoV at a farther distance (left) than smaller virtual content at a closer distance (right).

User-locked content

User-locked content creates few challenges for seated users; however, when users are standing or moving, it is a greater user challenge.

Virtual content can block a large portion of the travel path, particularly when it is placed in the center of the FoV.

Figure 5: Left: less of a user challenge because they are seated and not navigating. Right: obstacles such as a pet can enter the user’s travel path when they are hidden behind virtual content.

Peripheral user-locked content can also lead to difficulty detecting the edges of nearby obstacles (e.g., door frames) or other people.
Both size and position are important considerations for placement – anchoring to the inner edge (instead of the center) can prevent blocking user’s FoV with varied content sizes.

Figure 6: Top-down image of a user’s FoV within a HMD. The arrow represents the view straight ahead and the blue represents the bounds of the FoV. Red bars illustrate different panel configurations. Rotational offsets of bigger panels (left) can block more of the central FoV than a smaller panel (center). Offset of the panel based on the inner edge keeps the central FoV unblocked (right).

Design considerations for occlusion

Occlusion is a seemingly simple, but complicated problem that depends on factors outlined above. Importantly, users expect correct depth placement of virtual assets and that it remains within physical boundaries. Users also report a better experience when moving virtual content rebounds instead of passing through physical objects. We offer suggestions below to help mitigate user challenges in mixed reality:

Reduce the virtual content opacity when it is near the user, especially if not intended for interaction.

Figure 7: Left: the virtual object at full opacity blocks the view of objects behind it when it is near the user. Right: opacity is reduced so the user can identify potential obstacles behind the virtual content.

Keep user-locked content out of the user’s central FoV, minimize it when they are in motion, and avoid surprising the user when navigating (e.g., pop-ups, etc).
- Anchoring virtual content to the controller or user’s wrist can also allow users to intentionally raise it in front of them when they want to see it.
Scene allows users to define their activity space – this supports proper occlusion for marked physical objects and keeps virtual content within the defined walls.
Mixed reality identification can be enabled in settings if users want additional alerts near marked objects.
Enabling Depth API within Passthrough to render occlusions in real-time. Depth API provides a real-time depth map of the environment as seen from the user’s point of view; this enables the hiding of virtual content behind physical objects whenever it is farther away. It has the added benefit of working with dynamic objects (e.g., other people, pets) and objects that the user has not manually marked with Scene.

Next steps

For more health and safety API specific info and general considerations, see the following:

Did you find this page helpful?