Immersive Debugger AI Assistant

• Troubleshoot gameplay in-headset: “Where is this object?” or “List all the grabbable objects in the scene.” The agent searches the live scene and explains it in natural language.
• Spawn and modify objects by voice: “Duplicate this object and move it one meter away” or “Change the size of this object.” The agent instantiates objects, adds components, and tweaks transforms at runtime.
• Visual debugging: “Select this object and show me the bounding boxes.” The agent selects the object in the inspector and draws debug boundaries.
• Run your own workflows: if your AI agent is configured with project source access, custom [Tool] runtime tools, the official Unity Editor MCP, or external MCP servers, the agent can use all of them (for example, generate a full scene report and send it).

• Unity 6 LTS or later
• Meta XR Core SDK v205 or later on an existing project
• Voice SDK (com.meta.xr.sdk.voice): required — the feature uses it to start the voice conversation. Planned to migrate to other dictation solutions in a future version.
• A supported AI coding agent installed and authenticated. Registered backends: Claude Code, Gemini CLI, OpenAI Codex, and OpenCode
• A Meta Quest headset on the same accessible network as your computer

1. Install AI Agent Bridge. Select Install AI Agent Bridge. This is the component that lets your in-headset app reach out to an AI coding agent running on your computer. Wait for the green “AI Agent Bridge is installed” check.
2. Connect your AI coding agent. Select your agent tab. Either select Run Command for one-click registration, or copy the commands and run them in a terminal. Wait for “Connection verified”. This registers the MCP tools your AI agent needs to reach runtime information in your headset.
   Note: On Windows, allow the firewall setting in the same AI Tools window — this lets the headset connect to Unity on your local network.

1. In the Meta XR SDK Window, select Immersive Debugger. If the window isn’t visible, select Window > Meta > Meta XR SDK to open it. Select Enable (available at runtime in the Editor and on-device).
2. Open the AI Assistant section and select Enable.
   • A Remote Server bridges the headset and the Editor. The IP address, port, and access token must match on both the headset build and the Editor.
   • Configure push-to-talk input (controller button or pinch gesture) in this section.
3. Confirm the validation help box shows a successful connection to your agent client.

1. Ensure all rules pass in the Project Setup Tool, then build an APK and deploy.
2. During gameplay, toggle on the Immersive Debugger, then the AI Assistant Panel.
3. Verify the connection indicators are green.
4. Push-to-talk (hold the A button or pinch with the middle finger), speak a command, and release. The agent reply streams progressively (thinking, then tool calls, then reply).

• The agent connects through two HTTP servers (the Agent Bridge Remote Server and the Tools/MCP Bridge). Confirm both are running.
• Verify the IP address, port, and access token match exactly between the deployed headset build and the Editor. A mismatch shows “not connected” pill indicators in the panel.
• The headset and the Editor machine must be on the same accessible network.

• The Remote Server and MCP HTTP server listen on local ports that the headset connects to. Windows Firewall can silently block these inbound connections.
• Allow Unity (and the AI Agent Bridge) through Windows Defender Firewall for Private networks, and ensure the configured port is not blocked.
• If indicators stay red on Windows despite matching IP, port, and token, a firewall block is the most likely cause. Test by temporarily allowing the app, then add a scoped rule.

• The built-in demo token is shared by everyone, so it reaches its usage limit quickly. If you see rate-limit or quota errors, create your own token in wit.ai⁠ (free) and configure the feature to use that instead of the shared demo token.

• Responsiveness depends on the configured CLI agent, its setup, and the network connection.
• If your agent is launched with many skills and MCP servers loaded, both startup and every back-and-forth are slower (each exchange re-spins the agent). Trim the skills and MCP servers your agent loads, or expect longer waits per turn.

• Voice input needs the Voice SDK, a prerequisite (see Prerequisites). If it isn’t installed, install it and re-open the AI Assistant settings.
• If push-to-talk doesn’t trigger the recording indicator, check the Voice Input settings. The controller button and pinch gesture are both configurable and may have been remapped.