Teaching the Future: A Creative Recipe for XR Design

Since the launch of Meta Quest 3, Meta has invested in several paths and events designed to help new and seasoned developers explore development options for VR, mixed reality (MR), and 2D so they can seamlessly enter the XR space and reach their goals faster. Through hackathons, courses, tutorials, and grant programs, thousands of developers have used Meta’s capabilities and SDKs to approach the question of “What makes a great XR app?”

During this journey, two gaps have surfaced: 1) the need for formal academic courses for developers interested in XR, and 2) the need to focus on mixed reality design opportunities versus emphasizing technical challenges.

Stanford University has recently made a push to fill both of these gaps. In Spring 2024, Stanford’s Hasso Plattner Institute of Design (d.school) pioneered its first full XR design course called “Designing for Extended Realities.” At its core, the curriculum emphasizes a critical, intentional, and human-centered approach to XR design, where students start by finding and understanding real use cases before jumping into spatial prototyping.

Through hands-on workshops, critical discussions, and lectures across healthcare, education, entertainment, and other sectors, students from diverse backgrounds learn to identify user needs and create compelling VR/MR experiences. In addition to educating students on the tools needed to rapidly prototype and build XR proof-of-concepts, the curriculum was designed to balance ethical and market considerations, preparing students for the complex landscape of XR app development.

Co-lecturers Yicheng ‘YC’ Sun and Payam Tabrizian bring a blend of design insight, technical expertise, and entrepreneurial spirit to the course. YC, a seasoned lecturer at the d.school, also serves as a Senior Director at the global design and innovation consultancy, IDEO, where he leads its digital products and emerging technologies practices. Payam, Head of Maps at Uber, also brings a background in XR design leadership from Unity. Their combined expertise makes the course accessible and empowering for students of all backgrounds and technical levels.

As XR practitioners, YC and Payam observed that technological investment in XR has far outpaced the identification of compelling use cases. While XR has massive potential to solve people’s problems in healthcare, education, commerce, and other fields, it often risks becoming an end in itself rather than addressing real human challenges. During the course’s inaugural term, Meta and other industry partners presented their view of the opportunities to the class to help ensure that students focused on XR opportunities that closely align with real user needs.

During Meta’s session, students learned about the history of Meta’s VR/MR devices, the content types that currently have traction with users, and the different design considerations that make certain content more suitable for XR.

Blending industry information with an academic design-centered approach led to students investing time in both inspirational and high-utility XR applications. The course culminated in an exciting XR expo where student groups exhibited their immersive XR experiences to a crowd of hundreds, including potential funders, industry professionals, and a curious public. By providing students with practical, real-world considerations and hands-on development, these students are now poised to become XR industry leaders, drive innovation, and set new standards for what XR can achieve.

Designing an effective curriculum for an emerging field isn’t easy, but for academic institutions interested in exploring how to best prepare students to shape the XR industry and build the next wave of successful apps, YC and Payam’s recipe for XR education can be found below.

Now, let’s dive deeper into each ingredient to reveal how institutions can set their students up for success.

Before launching the “Designing for Extended Realities” course, YC and Payam applied design thinking principles. The process kicked off with a workshop at Stanford’s d.school to understand the needs and aspirations of students interested in the XR industry. This session engaged students in brainstorming and idea generation and featured a panel debate with industry leaders like Charles Migos and David Webster aimed at sparking a lively discussion about XR’s current challenges.

Subsequent follow-up sessions at IDEO, co-hosted by XR experts from Unity and IDEO further refined the course concepts. To ensure the curriculum remained relevant to real-world industry challenges, they consulted Winston Esposito from Meta, whose insights helped align course objectives with the latest use cases. Throughout the process, feedback loops between students, industry professionals, and peers continuously evolved the curriculum, future-proofing it for the dynamic XR landscape.

Unlike traditional XR courses, the d.school’s course prioritizes storytelling training and user empathy building in addition to technical skills. Students explored diverse use cases through brainstorming with users and ethnographic interviewing. Students were encouraged to keep weekly documentation of their contextual research, user visits (virtual and in-person), and the guerrilla product testing they conducted.

Students were also encouraged to experiment with narrative techniques and immersive environmental design. A notable example included a group that created an immersive golf training environment with Astroturf and golf clubs to replicate the skill-building frustrations a new golfer might have.

With no definitive textbook for designing in extended realities, the curriculum had to be created from scratch and involve intentional industry collaboration. Payam and YC leveraged their wide network, inviting organizations to provide guest lectures and discuss top industry questions and use cases they were keen to explore.

Students interacted closely with industry domain advisors who helped them frame problems, develop ideas, and craft compelling product stories. These partnerships were instrumental in helping students deepen their expertise in specific domains quickly and ensured their projects were grounded in real-world impact.

The course fostered a collaborative environment by intentionally pairing students from technical fields with those from business, entrepreneurship, and unexpected disciplines like medicine or law. From the outset, students were challenged to design and critique new XR product experiences that addressed real-world problems, culminating in a high-stakes pitch to a panel of guest experts.

These diverse student groups collaborated for weeks, often surprising instructors and guests with what they created. This approach not only taught students how to collaborate organically, but also how to leverage their strengths, whether in brainstorming, research, or user testing. As a result, they developed innovative solutions that would have been difficult to achieve within more homogenous groups.

Navigating the tension between low-fidelity and high-fidelity prototyping in XR is a key focus of the course. Spatial computing is unique in its broad range of prototyping methods, from 2D pen-and-paper sketches and see-through plexiglass simulations of mixed reality to role-playing as agents in virtual worlds.

Starting with low-fidelity prototyping helped build students’ confidence by allowing them to explore ideas quickly and without the pressure of technical perfection; however, it quickly became clear that design progressed fastest when students transitioned into mixed reality environments in their headsets. Tools like Bezi, ShapesXR, Unity, and Unreal played a crucial role in this transition, helping students move from initial concepts to more refined, high-fidelity prototypes. Though counterintuitive to our original approach, this insight will be a key focus in future iterations of the course.

Creating off-ramps for students to continue their projects was crucial in sustaining momentum, especially in an industry where many tech demos struggle to reach the market. Students were encouraged to see their final exhibitions as the starting point of their journey as XR designers. With our support, they explored various paths, whether through academic channels like submitting to conferences such as SIGGRAPH, entering startup accelerators, or using their projects as a foundation for further personal exploration.

For example, Koh Terai is continuing his XR projects for SIGGRAPH, Darynne Lee entered a startup accelerator with her fashion project, and Libby Ye used her EEG-based mindfulness XR experience as a foundation for a speculative future practice. These pathways not only helped students secure job offers and create significant portfolio pieces but also underscored the course’s commitment to nurturing long-term success in the XR field.

Created by Cindy, Maho, and Emily, Treehouse XR is a personalized, interactive academic paper reading experience that allows users to organize content by sections, record notes automatically, and stay focused.

Many students find reading academic papers to be overwhelming and isolating but with Treehouse XR, their experience is transformed into a collaborative and engaging activity that helps them learn, retain, and organize content while interacting with peers.

Created by Peter, Grace, and Minho, ProMotion XR is a motion-learning service in which users can see professional sports motions in XR and push themselves to the next level. Honing sports techniques often feels overwhelming for amateur athletes due to limited access to quality coaching and personalized feedback. With ProMotion XR, these athletes gain a personalized, immersive experience where they can visualize and mimic professional sports motions in real time, allowing for more effective practice and skill refinement.

With this tool, users gain access to expert techniques, helping them break down complex movements and apply them to their own training. This enhanced, interactive learning approach helps athletes feel more confident and motivated, ultimately improving performance and accelerating their progress. It transforms training from a repetitive, solo activity into a dynamic, goal-driven experience.

Created by Darynne, Anthony, and Gautam, VirtualOutfitters is an immersive tool designed to simplify the creative process for fashion designers by enabling seamless fashion prototyping and collaboration.

In a rapidly evolving industry like fashion, designers face challenges in quickly visualizing prototypes. Traditional methods are both time-consuming and resource-heavy, often leading to textile waste. With VirtualOutfitters fashion designers can create and visualize designs more efficiently, reducing both time and environmental impact.

Created by Trishia, Hunter, and Pannisy, ProtoML aids engineers by streamlining and simplifying 3D data processing for machine learning.

ProtoML allows engineers to easily import, clean, label, and process 3D data in real time, all within an immersive VR environment. This streamlined workflow improves data quality, leading to more accurate and reliable machine learning models. By simplifying 3D data handling, ProtoML enables engineers to focus on model development and innovation, speeding up the overall process.

As the XR field continues to evolve, the creative recipe on XR design education learned from this course will continue to guide future iterations, ensuring that the next generation of designers is ready to push the boundaries of what’s possible in XR. Please visit the Meta Horizon Developer Center and our mixed reality design documentation to learn more about what you can build with mixed reality and find best practices for designing delightful experiences that blend physical and virtual worlds.

To keep up with the latest news impacting developers building on Meta Horizon OS, follow us on X and Facebook, and subscribe to our monthly newsletter from your Developer Dashboard settings.