Mixed Reality, or MR, uses the physical environment around users to take your app and bring it to the world directly around them. It offers combined and unique experiences that blend virtual objects and the interaction with them with the real world around users.

Mixed Reality is a foundational pillar of the current and future Meta Quest headsets and is a key part of delivering next-level experiences to users in the Meta Quest ecosystem. This document addresses:

The Meta Quest Presence Platform contains SDKs and APIs tailormade to help build MR experiences on Meta devices. Blending design and use case considerations with these features will help you get started with your MR projects.

Before continuing, it’s important to distinguish between VR, AR, and MR:

Virtual Reality (VR) aims to fully immerse the user in a world built from the ground up for virtual experiences. In VR, a user wearing a Meta Quest headset is fully transported to a virtual place to enjoy virtual experiences. The following image shows a virtual creature in an entirely virtual space that a user can interact with.

Augmented Reality (AR) involves augmenting, or supplementing, the physical world around a user by superimposing virtual elements. AR utilizes the existing reality around a user to generate experiences that appear in the real world.

Mixed Reality (MR) is a combination and evolution of both VR and AR. It is inclusive of both technologies while also offering a new medium of interaction by allowing virtual elements to interact with both the user and the physical space around the user. It’s important to note that, because MR is inclusive of VR and AR, MR experiences can combine a varying amount of each technology to create an MR experience.

Static MR allows users to enjoy experiences while seated at their coffee table, in their bedroom, or even at their work desk.

Scenarios: With a combination of augmented reality (AR) and VR the flat surface of their desk can house a miniaturized strategy game, or host a collocated card or board game like Chess. Users can lean back and enjoy a large-scale digital event, like a concert, with a screen the size of their wall while their friends’ Avatars are collocated in the room with them.

With Static MR, developers can build around users being in a relatively fixed location. This location can be transformed and altered with the use of AR, and by displaying, and interacting with virtual objects.

Dynamic MR includes experiences that push the boundaries of the physical space where MR experiences can be contained.

Scenarios: Imagine a house transformed into a haunted house through MR, fending off an invasion breaking through the walls of a user’s bedroom, or active full body exercise experiences in a living room with a personal trainer.

These Dynamic MR experiences are built around engaging the user physically or involve movement between multiple rooms or locations.

2D & Classic styled MR, typically, includes static experiences based on interacting with 2D surfaces and planes.

Scenarios: A user can play a strategy game with units moving around their coffee table, a card game with their living room in full view, or a puzzle game with the pieces of the puzzle floating in places around their play area.

Users can enjoy a sense of awareness of their surroundings while enjoying these familiar experiences, and even navigate their play area by transforming these games and experiences with MR.

Some popular examples of bringing an otherwise 2D, seated or static virtual experience into MR, include:

One of the first steps to bringing a 2D experience into MR is to a user’s physical environment is using the cameras on a Meta Quest headset to view the surrounding environment. This can be accomplished with the Passthrough API.

At a base level, MR experiences start with a combination of virtual and augmented reality. Using Quest APIs like the Passthrough API allows you to begin combining these two technologies to bring familiar experiences to users in a new light.

The Passthrough API provides a real-time visualization of the world around a user while wearing a Meta Quest headset. The external facing cameras generate a passthrough layer of the user’s physical environment, which is then replaced with a rendition of the environment by an XR compositor.

Integrating the Passthrough API is the first step in combining virtual and augmented reality experiences that users can enjoy. Having a user mark a surface during passthrough setup allows you to build experiences around those reliably defined surface areas.

An example of this would be having a user define their desk, then using that defined surface to build MR experiences. Users could play a strategy game on their desk or enjoy a top down experience like classic arcade games. Using Passthrough in this way also enables users to interact with virtual objects placed on their defined surface area.

For more information about the Passthrough API, see the Unity, Unreal, or Native documentation.

Once a virtual element is blended into a user’s physical environment, convincing audio can help increase the plausibility of the MR experience. A user hearing a sound coming from an element in their physical environment can prompt curiosity, engagement, and the exploration of virtual elements in their space.

The Audio SDK supports spatialization features that transforms monophonic sounds and changes their origination point in the area around the user. Well designed audio creates an additional level of presence for the virtual elements in a user’s physical space and is a crucial component of persuading users to interact with the virtual elements in their environment.

You can transform familiar 2D experiences with MR, as interacting with virtual objects and companions brings an additional level of familiarity.

Scenarios: This can take the form of a player playing with a virtual companion hanging out in their room. This companion is aware of their living real-world space and can respond to voice and hand gestures. Such experiences can also include a virtual companion interacting with virtual objects, or virtual representations of real objects in virtual space.

A common throughline of MR experiences is the ability to candidly interact with virtual spaces, objects, or companions. After using Passthrough to identify surfaces and objects in their play space, these objects can dynamically interact with the area. This gives players a chance to interact with it through the various input forms available on Meta Quest headsets.

It’s important to clearly define how users interact with both virtual companions and objects in an MR experience. The different interactions available to a user ultimately defines their relationship to the virtual elements in the physical space around them.

The Interaction SDK contains a library of components that allow you to add controller and hand based interactions to your experience. By combining features like Passthrough and the components of the Interaction SDK, you can enable your users to interact with virtually placed objects or companions.

Placing a virtual object or a responsive virtual companion within a user’s space can serve to heighten their engagement with the experience you’ve created for them.

The first and most common form of interaction in VR and MR are through controllers. Integrating controllers enables experiences like petting a dog, wielding a sword, piloting a spaceship, or painting a beautiful 3D canvas in the room around them. The Meta Quest and Meta Quest Pro controllers offer eight combined buttons and triggers, plus 6DOF to simulate various interactions. Integrating controllers into your MR experience allows for precise inputs through buttons, triggers, and movements and haptic feedback when interacting with virtual objects or companions.

Using features like hand tracking allows users to use their own hands to interact with the virtual objects to do things like dribble a ball, roll dice, or solve a puzzle. Combining this with Passthrough can allow users to pick up and place virtual objects in their personal space.

The Voice SDK can allow your users to have a conversation with a virtual companion or give responsive commands that have recognizable and results in the world around them.

While solitary MR experiences can be extremely engaging, cooperative or competitive experiences with friends and rivals can make memorable experiences that can consistently bring users back.

Scenarios: Being able to share MR experiences with friends and rivals can consistently bring users back to play and compete again and again. Competing in a ping pong game, or playing with shared virtual objects in local multiplayer experiences through Passthrough can create some truly engaging and memorable moments.

Platform SDK is the basis for multiplayer on Meta Quest, and allows you to easily join users together in your experiences with APIs like Group Presence, App to App Travel, Leaderboards, and more.

Scenarios: By integrating Platform SDK in your MR app, users will be able to find and invite friends, travel to an experience together in a group, or invite friends to specific destination-based experiences within your app.

Integrating the Movement SDK allows you to track your user’s body, face, and eyes to further immerse users and their friends in a MR experience, allowing things like gestures and facial expressions to be communicated while using an app. It can also be used for fitness experiences to track the user’s body and guide them through exercises and other workouts.

Scenarios: Integrating the Movement SDK for your experience, allows users to more reliably interact in an MR environment. Face tracking allows users to truly embody a character or their avatar in a social environment with genuinely reflected facial expressions and body tracking allows for more immersive active experiences like fitness or party games.

Combining Platform SDK features with MR-specific tools like Shared Spatial Anchors can transform and augment a user’s local multiplayer experience. Shared Spatial Anchors can be created by a user and then virtually shared with other users collocated in the same physical space. This results in a shared virtual frame of reference for multiple users, who can share the same perspective in terms of where and how digital objects attached to shared anchors are placed.

Scenarios: Building on the previous examples, multiple users can create spatial anchors then share them with collocated friends to bounce a virtual ball off their living room wall. Any user that joins the room with these shared anchors would be able to join in the experience and interact with the virtual objects.

The Shared Spatial Anchors Sample provides a tutorial on incorporating them into a Unity project and can be used as a foundation to build upon for your Unity based MR apps.

After bringing virtual objects into a user’s play area through MR, the next step would be to physically incorporate the walls, floors, and objects around the user and bring them into the MR experience. Doing so, expands the concept of a play area into true room-scale MR experiences.

Scenarios: Imagine interacting with alien ships flying in from the walls, floors, or through the ceiling. Or building an amusement park that goes from the bedroom through the kitchen and around all the furniture in the living room.

One of the most common ideas present in the world of Mixed Reality is users interacting with something coming from their walls, floors, or ceiling. With the Scene API you can build an experience that includes all of these surfaces, plus objects like a user’s desk, couch, chair, windows, and more. With these in place, you can blend virtual content into the user’s physical space.

Scene API includes Scene Capture and Scene Model which both work together to build an up-to-date representation of the user’s physical space, including walls, furniture, and so on. These can be indexed and queried. The user’s scene model consists of Scene Anchors that consists of geometric components and semantic labels.

An example of a room-scaled MR game that uses scene capabilities, passthrough, voice, and hand tracking is the World Beyond Sample App.

With all of the building blocks in place, these SDKs and APIs can be combined to create an experience for your users that uniquely combines the virtual world and the physical world.

By integrating MR in a classroom through shared spatial anchors that reflect a historical setting and various historical artifacts, allows students to pick them up and interact with them to learn more about a specific time in history. Or those same colocated students could take notes in a physical notebook while virtually dissecting a frog for a science lesson on biology.

Your users can engage in mixed reality educational experiences that leverage SDKs and APIs like the Platform SDK and Spatial Anchors to build, interact with, and learn from persistent virtual objects in their classroom.

Transforming a living room or a bedroom into a concert venue, complete with a dynamic screen on a user’s wall, is just one of the ways MR can transform and innovate entertainment experiences.

Entertainment in Mixed Reality can be done in a wide variety of ways. Some additional examples include:

You can work on Fitness MR experiences by using Passthrough to bring a personal trainer’s avatar into someone’s living space to help them exercise. Additionally, objects like hand weights or other exercise equipment can have virtual equivalents that users can exercise with.

Future updates could include things like hand weights to incorporate a wider variety of exercises that users can partake in from the comfort of their own home.

Fitness and Wellness experiences can be further enhanced by incorporating the Movement SDK for body tracking, or hand tracking for accurate hand positioning tracking.

You can achieve and enhance productivity in MR in a variety of ways. For example, giving users a virtual office space, like Horizon Workrooms, can allow them to customize where they choose to work.

The Keyboard Tracking feature allows users to view their own keyboard in their virtual environment. By combining this with the hand tracking and Meta Avatars, you can allow users to fully express themselves even while in a productive, professional setting.