Dive into Learnings from Supernatural’s Mixed Reality Mode

Supernatural recently launched mixed reality mode to an incredibly positive community response. We were excited to embrace this aspect of the hardware and software stack, and many athletes have already expressed appreciation for the additional awareness of their immediate surroundings that mixed reality (MR) provides during periods of intense movement.

Even with the overwhelming response to MR mode, a core part of the Supernatural experience is the sense of immersion in a natural environment. Finding a way to maintain this immersion and provide the situational awareness of MR was an interesting technical challenge. Here’s how we approached it.

Most Supernatural workouts include movement around a complete 360 degree horizon. Over time, different portals will open dynamically to indicate an active ‘lane’ of targets. Prototyping demonstrated that allowing our virtual world to appear only around active lanes would provide a good balance of virtual world immersion directly ‘behind’ the focus, and real-world situational awareness beyond the focus.

Using an underlay layer allows us to control Passthrough visibility via framebuffer alpha. This is perfect for our use case as it provides per-pixel control over the transition area, enabling us to show the virtual environment, Passthrough, or a mixture of both.

To achieve this functionality, we divided rendering into three layers, segmented via Unity render queue:

Layer One: The environment and portals are rendered as normal.

Layer Two: The alpha icosphere is rendered to the alpha channel only. This is a highly tessellated ~10k vertex sphere that covers the framebuffer and writes an alpha mask, masking in pixels where the portals are visible, and masking out pixels where the Passthrough feed will appear. The design team has control over falloff on the masking via a texture curve (my idea of a dynamic, 70’s style wavy transition effect at the mask borders, while technically feasible, was politely declined by those with more refined aesthetic sensibilities…).

Layer Three: We render virtual objects that we always want to be visible. Since these are rendered after the icosphere, they can write an appropriate framebuffer alpha that overrides what the icosphere has laid down.

While conceptually quite simple, this approach generated some specific first and second-order challenges.

Browsing the extensive content catalog in Supernatural defaults to a UI paradigm with elements that are “large and far away.” This UI doesn’t work well in MR because it leads to dissonance with surfaces in the real world that can be much closer. To make content browsing in MR feel as comfortable as possible, we implemented UI scaling to move the virtual elements closer to the camera and thus run less risk of appearing to be ‘inside the wall.’

For similar reasons, we moved our coaches closer in warmup and cooldown sections. Even though we already streamed stereoscopic coach video at 2K, the reduced view distance of the coach led to some minor pixelation. To maintain our quality bar, we shifted to 3K which resulted in the coaches looking more crisp and clear up close. Support for HEVC embedded alpha allowed us to use the video frame area twice as efficiently, and would probably allow us to drop back down to 2K without a visual quality drop.

Running MR on Meta Quest 3 limits the CPU and GPU clock speeds available to apps. Dropping maximum available GPU speed by nearly 20% required us to think hard about how best to ensure we maintained a rock-solid frame rate, particularly given the additional rendering load that MR adds from the icosphere. Ultimately, we opted to drop framerate on Quest 3 from 90fps to 80fps in MR mode. This drop in framerate was barely perceptible for our blind test subjects and deemed a good tradeoff for additional GPU headroom. We remain hopeful that with ongoing optimization these MR-induced limits might be reduced in the future.

While the Passthrough photon-to-photon latency is impressive, it remains considerably greater than controller position latency. This results in a visible mismatch at the border between the virtual and Passthrough world when the controllers are moving quickly, and this disparity can be unsettling for some people. Of course, a fitness app with fast motion across a wide area will feel this latency challenge more than an app with slower movements.

We spent some time tweaking the border in an attempt to push the transition zone as far behind the range of motion as possible while still maintaining useful peripheral Passthrough. While we are happy with the results of our solution, the problem does still exist.

For some apps, one possible mitigation to the latency issue is allowing the controller to write a localized alpha ‘blob.’ A reasonable sized blob would dynamically mask in the virtual world around the controller, hiding the disparity.

Our team cares deeply about maintaining feature parity across the hardware ecosystem where feasible, so supporting MR on Meta Quest 2 was important to us. Unfortunately, the brief fade-out cycle that we use to hide some heavyweight processing between Supernatural environments isn’t possible in MR, which led to a visible Passthrough update stall on Quest 2 while the GPU ran a complex lighting convolution kernel.

Our solution was to re-architect this computation to amortize the processing over a configurable number of frames. We landed on ~5% of the kernel processing every frame for 20 frames. The resulting reduction in GPU load allows for a very smooth transition, even on Quest 2.

Using an underlay with alpha punch-through means you must carefully consider framebuffer alpha in your semitransparent layer three objects. Generally, this means extending your blend operations to specify a separate combination for the alpha channel that allows the icosphere-written alpha to persist. We ended up having to modify several shaders to accomplish this, and in some cases we had to write a replacement for Unity’s standard shader set that provided the alpha-specific control we needed.

Supernatural’s MR mode had a brisk journey from early prototype to release, a testament to both the internal excitement around supporting the feature, and the quality of the platform stack. Features this large invariably take an army, so I want to share a big thank you to everyone on the Supernatural team who pitched in to bring this feature to life!

Supernatural can be purchased from the Meta Horizon Store. To get the latest updates and developer news, including insights from developers like Supernatural, be sure to subscribe to our newsletter in your Developer Dashboard settings and follow us on X and Facebook.