Immersive Debugger AI Assistant

• Troubleshoot gameplay in-headset: “Where is this object?” or “List all the grabbable objects in the scene.” The agent searches the live scene and explains it in natural language.
• Spawn and modify objects with speech or text: “Duplicate this object and move it one meter away” or “Change the size of this object.” The agent instantiates objects, adds components, and tweaks transforms at runtime.
• Visual debugging: “Select this object and show me the bounding boxes.” The agent selects the object in the inspector and draws debug boundaries.
• Run your own workflows: use Unity CLI⁠ for persistent Editor or project changes and Meta XR Operator MCP tools for the running app. External MCP servers can supplement these tools, for example, to send a generated scene report.

• Unity 6 LTS or later
• Meta XR Core SDK v207 or later on an existing project
• [optional] Voice SDK (com.meta.xr.sdk.voice). Install it only for these speech configurations:
  • The Unity Editor on your computer, with or without Meta Quest Link
  • A desktop player
  • An Android player on Meta Quest 2 or Meta Quest Pro
  • Wit.ai as the speech backend
  Typed input does not require Voice SDK. An Android player on Meta Quest 3 or Meta Quest 3S uses system speech by default.
• A supported AI coding agent installed and authenticated. Registered backends: Claude Code, Gemini CLI, OpenAI Codex, and OpenCode
• For Quest APK setup and runtime tools, a Meta Quest headset connected to your computer with an authorized USB debugging connection. A configured local network connection is available only as a fallback for the Agent Bridge panel connection.

1. Step 1: Select your service provider. Choose the coding agent that you want to connect.
2. Step 2: Install and set up AI tools.
   • XR Operator card:
     1. Follow Getting started with Meta XR Operator and use the download, install, or update control displayed in this card.
     2. Select Run Command, or follow the manual instructions in a terminal. Wait for Connection verified.
     3. Select Activate XR Operator.
     4. For a Quest build, select File > Build Profiles, select Android, and enable Development Build. Connect and authorize the headset over USB, then select Set up headset.
        After every headset reboot, return to this card and select Set up headset again.
   • AI-powered XR tools card: Select Install AI Agent Bridge. Despite its current label, this control enables and initializes the bridge bundled with the Meta XR Core SDK. Wait for the green AI Agent Bridge installed check. With the headset connected and authorized over USB, select Configure ADB reverse.
3. Step 3: Allow local network connections (Windows Firewall). This Windows-only step applies when using the LAN fallback. Select Configure Windows Firewall.

• Install project-level skills (optional). These agent-specific skills provide Meta XR workflow guidance for creation tasks. You can skip this action if you only use Immersive Debugger runtime tools.
• Review Project Setup Tool. Select Open Project Setup Tool, fix outstanding AI-tools rules, and review the remaining project rules before building.

1. In the Meta XR SDK Window, select Immersive Debugger. If the window isn’t visible, select Window > Meta > Meta XR SDK to open it. Select Enable (available at runtime in the Editor and on-device).
2. Open the AI Assistant section and select Enable.
   • Enabling the AI Assistant causes the build to inject the microphone permission, system speech visibility queries, and the optional system keyboard manifest feature.
   • The build injects the LAN fallback settings automatically. Agent Bridge tries ADB reverse first, uses the LAN fallback when ADB reverse is unavailable, and retries the connection in the background.
   • For an Android player on Meta Quest 3 or Meta Quest 3S, leave Use Voice SDK for input off to use system speech. For any Voice SDK speech configuration listed in Prerequisites, install Voice SDK and enable Use Voice SDK for input.
   • Configure push-to-talk input in this section. The defaults are the right controller’s A button, the left controller’s X button, or a middle-finger pinch with either hand when hand tracking is active.
3. Confirm the status card shows AI Assistant Ready. This status verifies that the selected coding-agent configuration is valid and the Remote Agent Server is running. It does not verify XR Operator or the deployed headset connection.

1. Ensure all rules pass in the Project Setup Tool, then create a Development Build APK and deploy it. Quest runtime tools require a Development Build.
2. During gameplay, toggle on the Immersive Debugger, then the AI Assistant Panel.
3. Verify the connection indicators are green.
4. Send a prompt:
   • On a headset, select the keyboard button to open the system keyboard. If the enabled button does not open or reopen the keyboard, the System keyboard does not open troubleshooting guidance provides the v207 workaround.
   • In the Editor, on desktop, or through Link, use the text field with a hardware keyboard.
   • For speech, an Android player on Meta Quest 3 or Meta Quest 3S uses system speech by default. The other speech configurations listed in Prerequisites require Voice SDK. Hold the right controller’s A button or the left controller’s X button. With hand tracking active, pinch with the middle finger on either hand. Speak a command, then release.
   
   Prompt input is unavailable while the agent responds. The reply streams progressively through thinking, tool calls, and the final response.

• Confirm Agent Bridge is running.
• Agent Bridge tries ADB reverse first and retries the connection in the background.
• If ADB reverse is unavailable and Agent Bridge uses the LAN fallback, open the AI Assistant settings, select Detect IP and Use Project Token, and confirm the port matches between the headset build and the Editor.
• For the LAN fallback, confirm the headset and the Editor machine are on the same accessible network.

• This issue applies only to the LAN fallback. ADB reverse does not require an inbound Windows Firewall connection.
• In AI Tools, select Configure Windows Firewall to allow the headset to reach the local Agent Bridge port.
• If the indicators stay red, confirm the LAN fallback port is not blocked.

• For an Android player on Meta Quest 3 or Meta Quest 3S, leave Use Voice SDK for input off to use system speech. Built-in system speech currently uses en-US. If the runtime displays On-device speech model for en-US is downloading — try again shortly., retry shortly.
• For the other speech configurations listed in Prerequisites, install Voice SDK and enable Use Voice SDK for input.
• If push-to-talk doesn’t trigger the recording indicator, check the Voice Input settings. The defaults are the right controller’s A button, the left controller’s X button, or a middle-finger pinch with either hand when hand tracking is active. These controls can be remapped.
• Voice SDK quota and rate-limit errors apply only when you enable Use Voice SDK for input. If the demo token reaches its limit, create your own token in wit.ai⁠ and configure Voice SDK to use it.

1. Select Edit > Project Settings > Player > Android > Other Settings.
2. Set Application Entry Point to Activity.
3. If the project uses a custom AndroidManifest.xml, set the launch activity’s android:name to com.unity3d.player.UnityPlayerActivity and android:theme to @android:style/Theme.Black.NoTitleBar.Fullscreen.
4. Rebuild and deploy the app.

• Responsiveness depends on the configured CLI agent, its setup, and the network connection.
• Loading many skills and MCP servers can increase startup and response time.

• Confirm Meta XR Operator is installed and XR Operator is activated in AI Tools.
• For a Quest build, confirm Development Build is enabled.
• Connect the headset over USB, authorize USB debugging, and select Set up headset. Run Set up headset again after the headset reboots.