Develop

Meta XR Operator overview

Updated: Jul 27, 2026

Experimental feature

Meta XR Operator is an experimental component of the Meta XR Core SDK. Tool names, APIs, and behavior may change in future releases. Avoid depending on it in production apps.

Meta XR Operator is an OpenXR API layer that gives AI agents spatial context and control over a running Unity XR app. It ships as an optional, experimental component of the Meta XR Core SDK and works with any OpenXR-based Unity app without requiring changes to your app’s code.

The goal is to give agents rich spatial awareness. They can then perceive, understand, and act within virtual and mixed reality through a standard protocol and a set of tools.

The API layer sits between your app and the OpenXR runtime. It intercepts the runtime calls your app already makes. This exposes session state, tracking data, and input control to AI coding agents through the Model Context Protocol (MCP)⁠. Any MCP-compatible agent, such as Claude Code, can then observe and interact with your app. The same workflow runs in the Unity Editor, in Meta XR Simulator, or on a Meta Quest headset.

Example use cases

Meta XR Operator is built for agent-driven development and testing workflows, such as:

Reproducing and debugging issues by driving the app with natural language.

Running automated test passes without scripting each scenario.

Inspecting live session, tracking, and frame state while the app runs.

Prototyping agent-driven XR experiences and spatial reasoning.

What you can do with Meta XR Operator

Through MCP, an agent can use Meta XR Operator to:

Query the XR session state, system information, and per-frame data.

Navigate the Unity scene graph and inspect UI canvases, including interactable controls such as buttons, toggles, and sliders.

Read and control the head and controller poses, including smooth, interpolated movement.

Set controller input, such as buttons, triggers, grips, and thumbsticks.

Capture composited images that combine the physical environment and virtual content.

List spatial entities, such as rooms, walls, floors, tables, and other furniture, with their poses and semantic labels.

Perform spatial math, including vector and quaternion operations and Unity-to-OpenXR coordinate conversion.

Call custom, app-specific tools that you register from your own C# code.

How it works

Meta XR Operator runs a local MCP server inside your app and bridges it to your agent through an MCP proxy. The pieces fit together as follows:

Your Unity app runs with the Meta XR Operator API layer enabled.

The API layer intercepts OpenXR runtime calls and starts a local MCP server inside the app’s process.

An MCP proxy connects your AI coding tool to that server and discovers the available tools - including a set of pre-included tools as well as any custom tools your app registers. The proxy can be installed from the AI Tools panel, or found within the standalone downloadable bundle.

The agent calls tools to read state from, or send input to, the running app.

The MCP server is also reachable directly over HTTP with Server-Sent Events (SSE) at http://localhost:8720/sse. Connecting through the proxy is the recommended path; the direct endpoint is an advanced option.

Because the integration happens at the OpenXR layer, the same workflow applies whether you run in the Editor with Quest Link, in Meta XR Simulator, or on a Quest headset. When you run on a headset, you forward the port over ADB so the host machine can reach the on-device server.

Supported platforms

Platform	Runs with	Notes
Windows	Quest Link, Meta XR Simulator	Full tool support for desktop development.
macOS (Apple Silicon)	Meta XR Simulator	Editor and Simulator workflows.
Meta Quest	On-device builds	Requires the `android.permission.INTERNET` permission and ADB port forwarding to connect from a host machine.