Aug 18, 2026

Introducing Meta XR Operator: Close the Build-Test-Verify Loop for VR

Read time: ~8 minutes

It's no secret that agentic coding tools are rapidly changing how developers build software. In just a couple years, these tools were adopted to read code, write code, run tests, and fix bugs in a fraction of the time it once took.

But for VR development, the agentic loop breaks down at verification: your agent finishes writing code and goes blind because it can't see the app, press a button, or confirm that everything works.

To close that gap, we created Meta XR Operator. It's an OpenXR API layer, now available as an experimental component in the Meta XR SDK (v205)⁠, that lets any MCP-compatible AI agent see, navigate, and interact with your running VR app in Meta XR Simulator. Your agent can finally build, launch, screenshot, identify issues, fix them, and verify the fix landed, autonomously.

In this blog, we'll walk through how it works, what you can do with it, and how to get started:

How it works

Three workflows it enables

Real examples from early testing

Get set up in about five minutes

Current limitations and what's next

What's Under the Hood

Meta XR Operator sits between your app and the OpenXR runtime, intercepting existing runtime calls to expose XR state and input control via the Model Context Protocol (MCP). It pulls screenshots and scene data only when your agent requests them rather than streaming continuously, so your frame rate stays untouched. Point any MCP-compatible agent at the proxy and the tools show up automatically, with no app code changes needed.

With Meta XR Operator, your agent gets access to a range of capabilities that close the loop between writing code and confirming it works:

Vision: Screenshot the running app as rendered in Meta XR Simulator, seeing what you'd see in the headset.

Spatial awareness: Read room geometry (walls, floors, furniture) and reason about 3D positions.

Controller input: Press buttons, move thumbsticks, pull triggers, point and aim.

Movement: Set head position and orientation, look at specific objects or areas.

Scene inspection: Read the Unity scene hierarchy, check components, and cross-reference what the agent sees with what the code says.

UI navigation: Press buttons, work through menus, and navigate multi-step flows autonomously.

You can also register custom MCP tools from your own Unity C# code, so domain-specific automations like studio QA checks or custom build steps slot right in on day one. Visit the Meta XR Operator documentation to explore more details.

The development loop with Meta XR Operator. Your agent validates each change visually, routes failures back to a fix step, and advances to delivery only once the change passes.

How the Pieces Fit Together

There are two moving parts.

A runtime MCP server starts with your app and serves connections at localhost:8720. It runs inside your app's process, which is what gives it access to session and scene state.

An MCP proxy sits between your agent and that server. It's easy to overlook, but it matters because your app isn't always running: you stop Play mode, recompile, and start it again throughout a session. Without a proxy, every one of those transitions would break the agent's connection and force you to re-register the server. The proxy holds a persistent connection for the agent, manages the switch between online and offline, and picks up tool changes as they happen.

The proxy stays connected to your agent whether or not the app is running, so entering and exiting Play mode never breaks the session.

💡 Bring your preferred model:

Meta XR Operator is model-agnostic and works over the open MCP standard with any compatible agent. Through the AI setup flow in the Unity Meta XR SDK (v205 or later), you can quickly set up Meta XR Operator with some of the most popular agents like Claude Code and Codex.

What You Can Do With It

1. Rapid Iteration

Describe what you want to build, iterate or test. Your agent writes code, builds, launches in XR Simulator, screenshots to verify, and iterates. Each feature gets validated visually and spatially before moving to the next, so you spend your time reviewing outcomes instead of manually checking every step.

This workflow works best for incremental features with clearly visible results.

2. Streamlined Debugging

Describe the symptom. The agent launches the app, reproduces the issue visually, inspects the scene and code, identifies root cause, applies a fix, rebuilds, and proves the fix works with a new screenshot.

Examples:

A UI effect doesn't fire when a button is pressed. Your agent navigates to the button, clicks it, sees nothing happens, traces the handler, fixes it, and verifies the fix.

Targets don't respond to projectiles. Your agent detects missing mesh colliders, adds them, and verifies that collision works.

A gun falls through the floor. Your agent identifies gravity enabled without a floor plane, disables it, and verifies the object stays in place.

3. Natural-Language Test Automation

Describe what should happen in plain English. Your agent executes the scenario and reports pass/fail with screenshots, with no test scripts to write and no test infrastructure to maintain.

Examples:

"Shoot a target, verify it despawns, a new target spawns, and the score increments."

"Press each UI button and verify it triggers the expected visual change."

"Navigate the full menu flow from title screen through song selection to gameplay."

This workflow runs in Meta XR Simulator and works best for static interactions with deterministic outcomes. Moving objects and timing-dependent behavior may still be difficult to capture accurately.

Putting It Into Practice: Real Examples

During early testing with Beat Games and other teams building on the Meta XR SDK, agents running Meta XR Operator completed full build-test-verify cycles without human intervention. Note that these examples are from pre-release builds and may not reflect the final experience.

Building and Playtesting

When prompted to build interactive Tic-Tac-Toe in VR, the agent wrote the game logic, aimed the virtual controller at cells, pressed "A" to place pieces, and played through to a complete win/lose state. The agent also attempted to adjust the display of the game result. When four styling fixes had no effect, the agent ran its own diagnostic, set text to white as a control, and traced the root cause to a Unity script-reimport issue.

Catching a Bug That Scripted Tests Missed

An agent tested Beat Saber's menu flow from the title screen through solo mode, song selection, difficulty, and into gameplay. When a UI button moved, the agent adapted without prompt changes. It also caught a hidden UI overlap bug that scripted tests bypass entirely because they don't interact with the visual layer, surfacing something that may have previously required a human tester to identify.

Get Started with Meta XR Operator

Follow the steps below to get set up with Meta XR Operator in about five minutes:

Set up your Unity project with Meta XR SDK v205 and Meta XR Simulator.

Install the AI bridge from Meta XR Tools → AI Tools for Meta XR SDK.

Connect your AI tool by selecting your assistant tab and running the connection command.

Verify the connection and optionally install XR-specific skills in your project folder.

Watch the short video below to see just how easy it is to set up and start iterating.

Requirements:

Unity Editor (6000.0.66f2 LTS or newer)

Meta XR SDK v205⁠

Meta XR Simulator (v205 recommended)

An AI coding assistant that supports MCP (Claude Code, Cursor, Codex, Gemini CLI, OpenCode)

Windows recommended (macOS has known controller input limitations)

Once connected, open a demo scene, hit Play in Meta XR Simulator, and try a prompt: "List the buttons in the XR environment" or "Aim the left controller at the bottom button and press it."

🎯 Hit the ground running:

Import the Meta XR Operator sample from the Meta XR Core SDK Samples tab. You get two demo scenes: CubeShooter (a small shooter game, plus a broken version with intentional bugs for the agent to find and fix) and UIInteraction (multiple UI canvases with intentionally broken interactables and copyable prompts embedded in the scene).

The CubeShooter sample running in Meta XR Simulator. Here the agent uses a custom driver script to aim and fire at five targets.

Know the Limits

Meta XR Operator is an experimental release that we're actively improving. Understanding these current boundaries before you dive in will help you get the most out of the tool and avoid unexpected friction:

Hand tracking: Your agent currently uses a single aim-ray per hand with no per-finger data, so hand-first interactions are not yet supported.

Audio: Your agent has no access to audio, which means spatial audio cues, voice prompts, and sound effects won't factor into its verification.

Video and animation: Your agent works from screenshots only and cannot watch animations play out or assess motion quality.

Moving targets: Your agent operates too slowly for real-time interaction, so it works best with static, deterministic scenarios.

Subtle visual bugs: Your agent tends to assume things look correct unless something is obviously broken, so it may miss slight color mismatches or minor alignment issues.

Engine support: Meta XR Operator is currently integrated into Meta XR Core SDK for Unity. If you are using another game engine backed by OpenXR, you can download the standalone package and do your own integration.

Platform support: It's recommended to use Meta XR Operator with Meta XR Simulator for the best performance and feature set. You can also use it on Meta Quest with additional setup.

If you run into issues or have ideas for what should improve first, submit feedback through the Feedback Center in Meta Quest Developer Hub.

What's Next

Meta XR Operator gives your AI agent the ability to see, navigate, and verify your running VR app, closing the loop that kept VR developers stuck in manual iteration. The setup is fast, it drops into the stack you already run, and it doesn't require changing your app's code, so you can immediately start using it on a project this week.

Later this quarter, a follow-up post will go deeper on how Meta XR Operator reads and interacts with Unity scenes, including scene hierarchy inspection, component queries, canvas/UI navigation, and custom-tool registration. Keep an eye out and follow our socials below to be among the first to hear when it drops.

Get started with the documentation and download Meta XR Core SDK v205⁠ today.

For more developer updates, follow us on X⁠, Facebook⁠, and LinkedIn⁠, subscribe to our monthly VR developer newsletter, and join the conversation in the Meta Horizon Developer Forums⁠.