MIXED REALITY WITH SCENE MESH AND SCENE API

First Encounters

Unlocking engaging mixed reality gameplay with Scene Mesh and Scene Anchors

THEIR STORY

Revealing the potential of MR

First Encounters is a mixed reality (MR) experience that comes pre-loaded on every Meta Quest 3. Showcasing the possibilities of MR for new audiences, First Encounters transports players to a strange new planet where space puffians—speedy and pesky fluff balls—smash through walls and need to be captured before time runs out!

When designing First Encounters, the team knew it needed an MR app that 1) demonstrated the types of experiences MR can unlock on Meta Quest 3 and 2) delivered gameplay that felt physical, grounded, and had realistic interactions. To achieve this vision, they relied on extensive use of Scene API which helped the app understand a player space and integrate virtual elements naturally into the real world.

THEIR GOAL

Build expectations through seamless destruction

First Encounters is the first MR experience many Meta Quest users ever engage with. Understanding the importance of first impressions, the team aimed to fully utilize players’ physical space to authentically demonstrate how MR can seamlessly blend environments to create interactive and enjoyable experiences. They decided to create a game in Unity centered around a set piece and core mechanic in which puffians burst through players’ walls to reveal a new, virtual world. The goal was for players to shoot and capture puffians throughout their physical environment before their walls crack and burst completely. In the mayhem, puffians would showcase scene understanding capabilities that enable them to float and bounce around physical objects to escape capture—as if they are actually in the same space as the player.

THEIR SOLUTION

Extending Scene Anchors and Scene Mesh to open up worlds

To deliver functionality that understands players’ space and creates virtual holes on physical walls with low processing cost, the team deployed a combination of Scene Anchors, Scene Mesh, Passthrough, and a custom DestructibleMesh system.

The use of Scene Anchors enabled primitives such as walls, floors, and ceilings to be labeled and represented as simple planes. Querying these primitives provides the system with a basic breakdown of players’ space. Leveraging Scene Mesh expanded this breakdown with an even better approximation of physical space by providing scanned triangle meshes of a player’s room.

The next step was deploying a solution that enabled players to see and navigate their physical space while interacting with virtual objects. Applying a Passthrough shader to the material of the Scene Mesh, the user could see their room as normal—until puffians start punching holes in the mesh.

After evaluating existing solutions built into Unity that would allow for virtual holes to be punched in walls and ceilings, the team opted for a custom solution that allowed them to move the processing cost to the start of the game and increase visual quality. The result was the DestructibleMesh system, which uses Voronoi partitioning to sort the Scene Mesh triangles into submech chunks that can be given their own GameObjects within Unity and individually disabled as they are destroyed. The DestructibleMesh system was written to use the Unity Jobs System for parallelization and the Burst compiler for fast native performance and SIMD vector instructions.

THEIR SUCCESS

Selective solutions, universal fun

By expanding its use of Scene Anchors and Scene Mesh with the DestructibleMesh system, the team achieved seamless destruction mechanics that overcame processing costs concerns, limitations in available texture data provided by Scene Mesh, and visual constraints preventing objects from being deformed in Passthrough. The result is a fun, action-packed experience that not only opens players' walls but opens up their imaginations to the many possibilities of MR.

Within Unity, certain queries such as Collider.ClosestPoint() should be avoided with Scene Mesh. Behind the scenes, this method in particular uses Physics.ClosestPoint(), and neither method works with concave meshes. When it comes to concave MeshColliders, these methods will fail silently, returning the original point passed in. For certain use cases, Physics.ComputePenetration() offers a potential alternative approach when working with Scene Mesh.”

Alexander Dawson

Software Engineer, Meta

The DestructibleMesh system is being improved to support extensible operations that sort and modify triangles both as pre-processing and as real-time steps at runtime. Operations can be queued by the user, and multiple meshes will be able to be queued to be modified on the Jobs system. In addition to making Scene Mesh modification and Health & Safety features easy to implement for future projects, this will also allow experimentation with different VFX and more precise shattering in the room destruction visuals for future updates of First Encounters.”