This documentation walks you through the features and structure of the passthrough relighting (PTRL) sample. It shows you how your Unity project can be set up such that virtual lights and shadows in your Unity scene can interact with scene/space anchor objects such as the floor, walls, desks, and so on. This helps your virtual content blend with the real-world environment displayed in passthrough.

Once the application starts, you see lights and shadows interacting with scene objects. You control the movement of the character, Oppy, which projects shadows and highlights.

Those lighting effects are hidden by the Environment Depth to be occluded by real-world objects, similar to the rest of the virtual scene.

You can control Oppy’s movement using the left or right controller thumbstick. Moving the thumbstick up moves Oppy in a forward direction relative to the headset.

To make Oppy jump, press the A, X, or the grip buttons. Jumping while moving is also possible, and encouraged in order to jump on scene objects like desks. Holding the jump button makes Oppy jump higher.

A control panel is attached to the left controller and can be interacted with using a ray shot from the right controller and the right trigger to select. Through the panel, you can adjust the render parameters for highlights and shadows, toggle between scene objects and the scene mesh, and change passthrough brightness and depth check.

The experience implementation is in the sample as a single scene named PassthroughRelighting.

The main elements of the scene include:

Everything that is required to create highlights and shadows in passthrough is located in the MRUK package in the core folder:

The resources for the Oppy and flame character are separated into their own respective folders in the sample.

Import the MRUK package to your project and link the PTRLScenePlane and PTRLSceneVolume prefabs from the folder onto OVRSceneManager.

Open the OVRSceneManager component in your project to set the OVRSceneAnchors prefab to type plane, volume, and global mesh; these prefabs should have a material assigned to them that makes use of the PTRLHighlightsAndShadows shader, from Core/Shaders folder.

The sample only features integration with Scene API. This means a user has to perform a manual scene capture or complete Assisted Scene Capture (ASC) to capture a scene mesh. However, the material can be applied on arbitrary objects, for example, a floor-level plane.

Import the MRUK package to your project and add the MRUK prefab to your scene. Add an EffectMesh component, link the TransparentSceneAnchor material to it’s MeshMaterial field. Add an MRUKStart component and link the CreateMesh method of the newly created EffectMesh to the list of MRUKStart scene loaded event list.

In order to show highlights from more than one point light source, make sure that the desired amount of per-pixel light sources is specified in your project settings. For the BiRP the setting is called Pixel Light Count and it is in Project Settings > Quality.

For the URP in the Universal Render Pipeline Asset, in section Lighting, set Additional Lights to be Per Pixel, and set the Per Object Limit to the desired number.

The appearance of shadows depends on your project’s quality settings.

To set the correct quality for your project, go to Project Settings > Quality. Enable Shadow, set the Shadow Resolution, and set the Shadow Distance. The smaller the distance, the higher the shadow quality will be.

The PTRLHighlightsAndShadows shader computes highlights intensity from point light sources and marks shadow areas adjusting transparency. This is processed in the OpenXR compositor and blended with the Passthrough layer.

When a virtual object rendered with this shader is overlapping with its real physical counterpart, it creates the effect of relighting, lit with virtual light and shadowed by virtual objects.

The BiRP subshader in PTRLHighlightsAndShadows contains several passes:

The first ForwardAdd pass is for point lights, highlights, and additional directional light highlights. This pass runs for each light that is not considered the main light. It accumulates diffuse contribution, this time without multiplying any albedo component that a regular diffuse shader would have.

After that, ForwardBase pass handles the shadows of the main light. It outputs a black color multiplied by the shadow intensity set in the material.

Finally, a second ForwardAdd pass adds shadow contributions from additional directional lights.

There is also a URP subshader which implements the PTRL effect in a single pass. UniversalForward pass computes light contribution and shadows from all light sources.

A common alternative to real-time shadows is blob shadows. Those are simple blots of color that do not take into account the geometry of the object and are more performant. The Oppy entity contains a BlobShadow game object that enables the technique. BlobShadow objects are designed using the Projector component, along with a specific material and shader that can be found in the corresponding folder.