Develop

Immersive Debugger AI Assistant

Updated: Sep 30, 2026

Experimental feature

The Immersive Debugger AI Assistant is an experimental feature. How the tool works with AI, its dependencies, and the setup flow may change in future versions.

Overview

The Immersive Debugger AI Assistant adds an in-VR chat panel to the Immersive Debugger. Use voice or typed prompts to inspect and change your running Unity scene without leaving VR.

In-headset AI Assistant panel mid-conversation

How it works

The Meta XR Core SDK gives your coding agent a debugging-focused system prompt. Meta XR Operator provides the runtime tools, and AI Agent Bridge connects the app to the CLI coding agent on your computer.

Use cases

Troubleshoot gameplay: Ask where an object is or list objects in the scene.
Change the running scene: Spawn objects, add components, or adjust properties and transforms.
Debug visually: Select objects and draw debug boundaries.
Run custom workflows: Use Unity CLI⁠ for persistent project changes, Meta XR Operator for runtime changes, and external MCP servers for other integrations.

Prerequisites

Before proceeding with this tutorial, complete the setup steps outlined in Set Up Unity for XR development to create a project with the necessary dependencies.

In addition, you need:

Unity 6 LTS or later
Meta XR Core SDK v207 or later on an existing project
[optional] Voice SDK (com.meta.xr.sdk.voice). Install it for voice input in the Unity Editor, through Meta Quest Link, in a desktop build, or on Quest 2 and Quest Pro. It is also required to use Wit.ai on Quest 3 and Quest 3S. Without it, Quest 3 and Quest 3S use system dictation. Typed input does not require Voice SDK.
A supported AI coding agent installed and authenticated. Registered backends: Claude Code, Gemini CLI, OpenAI Codex, and OpenCode
For Quest setup and runtime tools, connect the headset to your computer and authorize USB debugging. Agent Bridge can use the local network as a fallback.

Set up the AI Assistant

Setup has two parts: first configure Agent Bridge and Meta XR Operator through the AI Tools Setup Wizard, then enable the Immersive Debugger AI Assistant.

Part 1: Configure AI tools and your headset

In the Meta XR SDK Window, select AI Tools. If the window isn’t visible, select Window > Meta > Meta XR SDK.

In Unity 6000.3 or later, you can pin the SDK menu to the toolbar. Right-click the Unity toolbar and select Meta XR SDK > SDK menu.

Step 1: Select your service provider. Choose your coding agent.
Step 2: Install and set up AI tools.
- XR Operator card:
  1. Use the displayed control to download, install, or update Meta XR Operator.
  2. Select Run Command, wait for Connection verified, then select Activate XR Operator.
  3. For a Quest build, enable Development Build under File > Build Profiles > Android. Connect the headset over USB, authorize debugging, and select Set up headset. Repeat Set up headset after each headset reboot.
- AI-powered XR tools card: Select Install AI Agent Bridge. Despite its current label, this control enables and initializes the bridge bundled with the Meta XR Core SDK. Wait for the green AI Agent Bridge installed check. With the headset connected and authorized over USB, select Configure ADB reverse.
Step 3: Allow local network connections (Windows Firewall). On Windows, select Configure Windows Firewall if you need the LAN fallback.

Install project-level skills (optional). These agent-specific skills provide Meta XR workflow guidance for creation tasks. You can skip this action if you only use Immersive Debugger runtime tools.
Review Project Setup Tool. Select Open Project Setup Tool and resolve its rules before building.

AI Tools window with service-provider selection and setup controls

Part 2: Enable the Immersive Debugger AI Assistant

In the Meta XR SDK Window, select Immersive Debugger, then select Enable.
Open the AI Assistant section and select Enable.
Before building, match AI Assistant > AgentBridge Remote Server > Server Port to Meta XR > AI Agent Bridge > Advanced > Remote Server > Port in Unity Preferences/Settings.
Choose your input:
- On Quest 3 and Quest 3S, leave Use Voice SDK for input off to use system dictation.
- To use Voice SDK, install the package and enable Use Voice SDK for input.
- Typed input does not require Voice SDK.
Confirm the status card shows AI Assistant Ready. It checks Agent Bridge, its token, agent validation, and the Remote Server. Verify the port match in step 3, and check XR Operator and headset connectivity separately.

Immersive Debugger AI Assistant connection and voice-input controls

Deploy and use on your headset

Resolve the Project Setup Tool rules, then build and deploy a Development Build APK.
In your app, turn on the Immersive Debugger and open the AI Assistant Panel. Verify that the connection indicators are green.
Send a prompt:
- Select the keyboard button to type. If the keyboard does not open or reopen, follow System keyboard does not open.
- In the Editor, a desktop build, or through Meta Quest Link, use the text field with a hardware keyboard.
- For speech, use the bindings shown in the panel or configured under Voice Input. The defaults are A on the right controller, X on the left controller, or a middle-finger pinch with either hand. Speak, then release.
You cannot send another prompt until the response finishes.

Troubleshooting

Connection lost or indicators not green

Open Meta XR SDK Window > Immersive Debugger > AI Assistant and check the status card.
If the status card shows Token Mismatch, expand AgentBridge Remote Server and select Use Project Token. If it shows Invalid Connection Settings, set Server Port to a positive value.
If it shows Remote Server Not Running, open Unity Preferences/Settings > Meta XR > AI Agent Bridge > Advanced > Remote Server, then select Start Server.
For other states, follow the guidance on the card. Wait while it shows Validating AgentBridge....
Confirm AI Assistant > AgentBridge Remote Server > Server Port matches the Port in Unity Preferences/Settings under Meta XR > AI Agent Bridge > Advanced > Remote Server. The values must match for both ADB reverse and LAN.
After an Editor LAN address change, select Detect IP. After an Editor Port change, update the AI Assistant Server Port and restart the Remote Server when prompted. Rebuild and redeploy after either change or a token change.
If ADB reverse is unavailable, Agent Bridge falls back to LAN. Confirm that the headset and computer are on the same accessible network.

Windows firewall blocking the connection

This issue applies only to the LAN fallback. ADB reverse does not require an inbound Windows Firewall connection.
In AI Tools, select Configure Windows Firewall to allow the headset to reach the local Agent Bridge port.
If the indicators stay red, confirm the LAN fallback port is not blocked.

Voice input not working or reporting an error

On Quest 3 and Quest 3S, leave Use Voice SDK for input off to use system dictation. If the en-US speech model is missing, the AI Assistant requests it. Retry after the download completes; the panel does not show download progress.
For the other environments listed in Prerequisites, install Voice SDK and enable Use Voice SDK for input.
If push-to-talk doesn’t trigger the recording indicator, check the Voice Input settings and confirm the configured controls.
Voice SDK quota and rate-limit errors apply only when you enable Use Voice SDK for input. If the demo token reaches its limit, create your own token in wit.ai⁠ and configure Voice SDK to use it.

System keyboard does not open

For a v207 Quest build that uses GameActivity, use this workaround if the keyboard does not open or reopen. If the button is disabled, wait for the AI response to finish. You can also use voice input or Meta Quest Link with a hardware keyboard.

For local development and testing:

Select Edit > Project Settings > Player > Android > Other Settings.
Set Application Entry Point to Activity.
If the project uses a custom AndroidManifest.xml, set the launch activity’s android:name to com.unity3d.player.UnityPlayerActivity and android:theme to @android:style/Theme.Black.NoTitleBar.Fullscreen.
Rebuild and deploy the app.

The AI Assistant already adds oculus.software.overlay_keyboard; do not add it manually.

Follow the entry-point guidance before shipping

Project Setup Tool entry-point guidance depends on the installed Unity version. Unity 6000 versions affected by GameActivity ANR behavior recommend Activity, while other supported versions require GameActivity. Before shipping, follow the application-entry rule or recommendation shown by Project Setup Tool for the installed Unity version. Select **Fix All** to apply the recommended entry point and regenerate the manifest. Review the listed changes before selecting **Fix All**, because it can apply other outstanding Project Setup Tool rules.

Runtime tools are unavailable

Confirm Meta XR Operator is installed and XR Operator is activated in AI Tools.
For a Quest build, confirm Development Build is enabled.
Connect the headset over USB, authorize debugging, and select Set up headset again after each reboot.

Key concepts

Concept	Notes
Dormant by default	Agent Bridge and the runtime tools start only when explicitly enabled.
Your agent, your usage	The feature uses your own CLI agent under your own subscription. The Meta XR Core SDK supplies the system prompt, AI Agent Bridge transports prompts and responses, and Meta XR Operator provides runtime tools.
Agent Bridge abstraction	Switch between supported clients without changing your workflow.
Meta XR Operator runtime tools	The agent queries and changes the scene, game objects, and components through runtime tools.
Agent Bridge panel connection	The panel tries ADB reverse first and uses LAN only as the Agent Bridge fallback. These paths transport Agent Bridge messages, not Meta XR Operator runtime-tool traffic.