Building Mixed Reality Experiences with Presence Platform

The metaverse will open up new possibilities for building a wide range of experiences and businesses, limited only by your imagination—from virtual offices where coworkers from around the world can come together to collaborate on a shared whiteboard, to mixed reality adventures with magical creatures appearing in your living room. While the metaverse may still be five to 10 years away, we’ve already made a lot of progress towards our vision, building the technologies and capabilities that enable you to build immersive new experiences. These tools can help you experiment and explore compelling use cases while the metaverse is still taking shape and start shipping these experiences on Meta Quest. And now, we're excited to share our next steps.

With the release of the v40 SDK, the full set of Presence Platform capabilities that we revealed at Connect last fall are now available. With the general availability of Spatial Anchors and Scene, you can build and ship apps using all of our Presence Platform capabilities. Presence Platform—which includes these APIs, as well as, the Interaction and Voice SDKs, and our newly advanced Hand Tracking API—features a suite of machine perception and AI capabilities that allow you to build more realistic mixed reality, interaction, and voice experiences that seamlessly blend virtual content with the physical world. Go hands-on with all of these capabilities in our demo app The World Beyond⁠, available now on App Lab. We’ll add even more capabilities to Presence Platform—including richer social presence—this fall.

We’ll also be taking mixed reality experiences to the next level with color passthrough technology on Project Cambria⁠, our next-generation VR headset. Project Cambria is packed with new technologies and will be a major leap forward for VR hardware when it releases later this year. When you build an app with Presence Platform for Meta Quest 2, it will also work on Project Cambria once it ships.

We’re adding Scene to our growing list of Presence Platform capabilities. Announced at Connect 2021, Scene provides geometric and semantic representations of people’s spaces and works with Passthrough and Spatial Anchors to help you create room-scale mixed reality experiences. Scene is a developer-friendly representation of the real world which allows you to quickly build complex and scene-aware experiences that have rich interactions with the user’s environment. Apps can be aware of objects in physical space, like your couch or desk, and know how and when to blend virtual objects with the physical environment so they can co-exist. For example, you can attach a virtual screen to a physical wall or have a virtual character walk behind your couch with realistic occlusion.

With v40, we’re also releasing the general availability of Spatial Anchors, which are world-locked frames of reference that enable apps to place and orient virtual content in a physical space, persisted across sessions. For example, a virtual screen can be carefully positioned in front of your couch or you can display your favorite 3D art in your living room.

With the Scene, Spatial Anchors and existing Presence Platform capabilities, we’re enabling fast creation of mixed reality and laying the groundwork for even more immersive experiences on Quest.

Schell Games has been working on a not-yet-available, one-level demo for Meta Quest 2 called I Expect You To Die: Home Sweet Home, which uses Passthrough and Scene to blend the digital elements of a spy’s life into your home as you explore the immersive world of espionage. “Just floating virtual objects over a real scene isn’t very interesting, but by using Spatial Anchors, we can connect virtual objects to real ones in ways that are really compelling,” says Jesse Schell⁠, the CEO of Schell Games. “The promise of mixed reality is to bring magical virtual objects into your real life, and Presence Platform really makes that happen.”

Fish Under Our Feet, a short experience that Resolution Games has developed, uses Presence Platform’s capabilities to recreate the wintery pastime of ice fishing, all from the warmth and comfort of your own living room. “Since you see your world around you, you can afford to be more ambitious in your movement, which opens up a lot of opportunities to take games into more places than just your living room,” says Mathieu Castelli, the Chief Creative Officer at Resolution Games.

Gravity Sketch⁠ uses Presence Platform tools to help designers, engineers, and key decision-makers in teams collaborate more effectively. “Many teams use Gravity Sketch for their day-to-day work building physical or digital products that exist within a broader real-world context,” says Gravity Sketch Co-Founder and CEO Oluwaseyi Sosanya. “It’s empowering to now have the ability to see their work in context without the need to create physical prototypes and review ideas collaboratively regardless of their location.”

We’ve released a demo app for Meta Quest 2—The World Beyond—that incorporates all of our current Presence Platform capabilities. We previewed The World Beyond at Connect last year, and we’re excited to let the community get hands-on with the experience. In it, you’ll interact with virtual objects as if they were in the same physical space as you—your hands will occlude objects you grasp, and our virtual pet, Oppy, will share your space, respond to your voice, and be occluded by furniture as she walks around. You can bounce virtual balls off your real walls and furniture—and then watch as the real world gives way to a virtual forest. We hope it inspires you to create with these tools and demonstrates the far-reaching potential of blending elements of real environments into virtual experiences.

But The World Beyond is just a proof-of-concept, and there’s a lot more work to come before we reach our metaverse vision—where this type of mixed reality experience is ubiquitous, high fidelity, and truly transformative.

For one thing: We’ll need more advanced hardware…

Project Cambria will be the first in our new class of high-end VR headsets. It features new sensors, including high-resolution outward-facing cameras, that enable our first-ever color Passthrough experience. It also utilizes advanced reconstruction algorithms along with an IR depth projector that help recreate the depth and perspective of the space around you. Together, these technologies are key components that will enable you to build more realistic mixed reality apps. Here’s⁠ a sneak peek of the future of mixed reality—in full color!—via The World Beyond on Project Cambria.

Don’t worry—getting The World Beyond to take advantage of Project Cambria’s bells and whistles didn’t require a bunch of extra effort. When you build an app with Presence Platform for Meta Quest 2, it will also work on Project Cambria once it ships—and in higher fidelity and with color Passthrough for a richer experience, thanks to its advanced hardware specs.

“Presence Platform is our way to take these extremely challenging machine perception problems and encapsulate them underneath developer-friendly APIs which hide all the complexity,” says Meta Product Manager Sarthak Ray. “So if you’re trying to create content using color Passthrough, you don’t have to deal with things like depth reprojection or color alignment. Developers will get an easy-to-use API that will do this work for them.’”

While we’re still developing Project Cambria, even this early demo of The World Beyond and how it’s improved by the new hardware shows how much potential there is for mixed reality to open up unique experiences. The experiences we’ll begin to see on Meta Quest are the foundation for what’s possible in the metaverse tomorrow, and we hope you’ll continue to build alongside us.