Customize Passthrough

• Texture opacity: Change the opacity of the passthrough image (independent of the edge rendering) using textureOpacityFactor.
• Edge rendering: Runs an edge detection algorithm on the passthrough image and superimposes the result on top of it. Set XrPassthroughStyleFB::edgeColor::a to 0 to disable edge rendering.
• Color reproduction: Customize the color reproduction of the passthrough image by adding one of the following types into the next chain of XrPassthroughStyleFB:
  • XrPassthroughColorMapMonoToMonoFB: Convert passthrough image to grayscale (if applicable) and map each input value to a new grayscale value based on a look-up table.
  • XrPassthroughColorMapMonoToRgbaFB: Convert passthrough image to grayscale (if applicable) and map each input value to a new RGBA value based on a look-up table.
  • XrPassthroughBrightnessContrastSaturationFB: Adjust the brightness, contrast, and saturation of the passthrough image.
  • XrPassthroughColorMapLutMETA: Map each input color (RGB) to a new RGB(A) output color based on a look-up table (color LUT).
  • XrPassthroughColorMapInterpolatedLutMETA: Map each input color (RGB) to a new RGB(A) output color based on the linear interpolation between two color LUTs. This option can be used to smoothly transition between two color LUTs.

• Color scale is applied to the entire layer contents, including edge rendering, while the corresponding style features only affect the passthrough images.
• When Surface-projected passthrough is used, the style’s textureOpacityFactor is applied individually to each surface/triangle, which can lead to visible overlaps between surfaces. On the other hand, the alpha value of color scale is multiplied with the passthrough layer after it has been fully rendered (flattened), leading to compositing that’s akin to layering in 2D graphics software.

// Define the color LUT data
constexpr size_t kResolution = 32;
constexpr size_t kSizeOfRgbPixel = 3;
constexpr float kNormalizationFactor = 255.0f / (kResolution - 1);
std::vector<uint8_t> lutData;
lutData.reserve(kResolution * kResolution * kResolution * kSizeOfRgbPixel);
for (size_t b = 0; b < kResolution; ++b) {
  for (size_t g = 0; g < kResolution; ++g) {
    for (size_t r = 0; r < kResolution; ++r) {
      lutData.push_back(std::min(r * kNormalizationFactor, 255.0f));        // red
      lutData.push_back(std::min(g * kNormalizationFactor * 0.3f, 255.0f)); // green
      lutData.push_back(std::min(b * kNormalizationFactor, 255.0f));        // blue
    }
  }
}

// Create the color LUT
XrPassthroughColorLutMETA lutHandle;
XrPassthroughColorLutCreateInfoMETA createInfo{XR_TYPE_PASSTHROUGH_COLOR_LUT_CREATE_INFO_META};
createInfo.channels = XR_PASSTHROUGH_COLOR_LUT_CHANNELS_RGB_META;
createInfo.resolution = kResolution;
createInfo.data.buffer = lutData.data();
createInfo.data.bufferSize = lutData.size();
XrResult result = xrCreatePassthroughColorLutMETA(passthroughFeature, &createInfo, &lutHandle);
if (XR_FAILED(result)) {
  LOG(“Failed to create a passthrough color LUT.”);
}

// Apply the color LUT to a passthrough layer
XrPassthroughColorMapLutMETA colorLutStyle{XR_TYPE_PASSTHROUGH_COLOR_MAP_LUT_META};
colorLutStyle.colorLut = lutHandle;
colorLutStyle.weight = 1.0f;

XrPassthroughStyleFB passthroughStyle{XR_TYPE_PASSTHROUGH_STYLE_FB};
passthroughStyle.textureOpacityFactor = 1.0f;
passthroughStyle.next = &colorLutStyle;

XrResult result = xrPassthroughLayerSetStyleFB(passthroughLayer, &passthroughStyle);
if (XR_FAILED(result)) {
  LOG(“Failed to set style on a passthrough layer.”);
}

C_out = (1 - weight) * sourceColorLut[C_in] + weight * targetColorLut[C_in]

• For high resolution color LUTs (especially 64), xrCreatePassthroughColorLutMETA can take a few milliseconds and is best done in advance, outside of time-critical sections.
• An instance of XrPassthroughColorLutMETA instances occupy an amount of memory that is proportional to the LUT data size. Call xrDestroyPassthroughColorLutMETA to free this memory.
• Color LUTs can be applied to multiple passthrough layers by referencing it in xrPassthroughLayerSetStyleFB calls on multiple layers. Doing so reduces the memory usage (compared to creating a separate XrPassthroughColorLutMETA instance per layer).

• Setting the layers’ opacity values to create a full-screen blend between passthrough and VR by using styling for passthrough layers or XR_KHR_composition_layer_color_scale_bias for any layer type.
• Masking passthrough layers, either with the application’s alpha channel or with a separate projection layer containing only the alpha mask.
• Using surface-projected passthrough.

• Draw the alpha mask to a separate projection layer and submit it between the passthrough layer and the application layer:
• Using XR_FB_composition_layer_alpha_blend, set the blend factors for the alpha mask layer such that the color values remain unchanged and the alpha values are overwritten:

alphaBlend.srcFactorColor = XR_BLEND_FACTOR_ZERO_FB;
alphaBlend.dstFactorColor = XR_BLEND_FACTOR_ONE_FB;
alphaBlend.srcFactorAlpha = XR_BLEND_FACTOR_ONE_FB;
alphaBlend.dstFactorAlpha = XR_BLEND_FACTOR_ZERO_FB;

• Using the same extension, set the blend factors for the passthrough compositor layer such that the source color is multiplied with the destination alpha value:

alphaBlend.srcFactorColor = XR_BLEND_FACTOR_DST_ALPHA_FB;
alphaBlend.dstFactorColor = XR_BLEND_FACTOR_ONE_MINUS_DST_ALPHA_FB;
alphaBlend.srcFactorAlpha = XR_BLEND_FACTOR_ONE_FB;
alphaBlend.dstFactorAlpha = XR_BLEND_FACTOR_ZERO_FB;

• Create a passthrough layer and place it before the application layer in the layers list in xrEndFrame:
• After drawing the virtual world to the framebuffer, make an extra draw pass which clears out (or reduces) the alpha values in regions where passthrough should be visible.
  • For example, you can render a shape with a shader that returns an alpha value of 0 and a blend mode which causes the alpha value in the framebuffer to be replaced with the shader output and the color to remain unchanged.
• The application layer is commonly submitted as premultiplied by alpha. This leads to the source color values not getting attenuated by the alpha value in regions where the alpha value has been reduced in the previous step. There are two alternative solutions to this problem:
• Switch the application layer to unpremultiplied alpha by setting XR_COMPOSITION_LAYER_UNPREMULTIPLIED_ALPHA_BIT (and, if XR_FB_composition_layer_alpha_blend is used, picking the appropriate blend factors). This changes the interpretation of the colors for the entire layer, so the application needs to ensure that all color values in the framebuffer are not premultiplied with alpha.
• In the previous step, where the alpha value is modified, also modify the color value to make it reflect the premultiplied state. For example, multiply both color and alpha with the desired (inverse) passthrough opacity.

• Request the extension XR_FB_triangle_mesh when creating the OpenXR instance in the app code (see Creating an Instance and Session).
• When creating the passthrough layer, set XrPassthroughLayerCreateInfoFB::purpose to XR_PASSTHROUGH_LAYER_PURPOSE_PROJECTED_FB to activate surface-projected passthrough for the layer. The layer will not be visible until geometries are created for the projection surface.
• Create a triangle mesh object using xrCreateTriangleMeshFB.
• Add an instance of the triangle mesh to the projection surface using xrCreateGeometryInstanceFB.