Develop

Select your platform

Immersive Web SDK — AI-Assisted Development Tooling

Updated: Feb 27, 2026

What Is the Immersive Web SDK?

Immersive Web SDK (IWSDK) is Meta’s open-source framework for building WebXR experiences. It ships as a set of npm packages under @aspect/iwsdk-* and is available at github.com/facebook/immersive-web-sdk⁠ (MIT license) with documentation at iwsdk.dev⁠.

Core architecture: Three.js rendering + a high-performance Entity Component System (ECS) powered by ELICS + Vite-based tooling.

What the SDK handles out of the box:

Capability	Detail
Grab interactions	One-hand, two-hand, distance grab
Locomotion	Teleport, smooth movement, snap turn
Physics	`Havok` integration, collisions, rigid bodies
Hand tracking	Full articulated hand input and gestures
Spatial UI	VR panels with CSS-like styling (UIKitML)
3D audio	Spatial sound positioning
Scene understanding	AR plane/mesh detection
XR emulation	Browser-based WebXR emulation via IWER (no headset required)

Developer workflow:npm create @iwsdk scaffolds a complete project. Save a file, the browser hot-reloads, and the emulated XR session updates — no compile step, no APK build, no headset needed during development.

Launched at Meta Connect 2025. In the first two weeks: 50+ GitHub stars, 100+ clones, 4 forks. As of February 2026, average of 1M users engage with WebXR experiences each month across the ecosystem.

Overview

This update adds AI-assisted development tooling to IWSDK, enabling an AI agent (Claude Code) to not just write code for an XR application, but to see, operate, inspect, and debug the running experience in real-time. The result is a closed-loop development workflow where a developer describes what they want in natural language and the AI builds it — iterating autonomously by observing its own output.

We call this capability “Vibe Coding for VR.”

One-line summary: An AI agent can now build, test, and debug a WebXR experience end-to-end — without the developer writing a single line of code.

Target users

Persona	Description	How AI tooling helps
Non-technical creators	PMs, designers, artists, QA with ideas but no XR development experience	Describe an experience in words → get a working VR app
WebXR developers	Developers building with IWSDK who want to move faster	AI handles boilerplate, debugging, asset integration
Rapid prototyping teams	Teams that need to quickly validate XR concepts	Go from idea to shareable prototype in hours, not weeks
Enterprise builders	Organizations evaluating XR for training, visualization, collaboration	Low barrier to entry; no specialized XR engineering team needed

Validation

12+ people across Meta — product managers, data scientists, QA engineers — built full VR experiences in 1–2 hours with zero prior XR development experience during internal testing:

Beat Saber clone (PM, 2 hours)

Claw machine game (Data Scientist, 2 hours)

Vehicular combat experience (Browser Engineer, 1 hour)

Arcade pong game (Engineer, 2 hours)

Shooting gallery (QA)

Ski simulator (Engineer, no prior WebXR knowledge)

Feature specification

The AI tooling is delivered through three integrated systems that together form a closed development loop.

Claude Code (the AI agent) connects to an MCP server running inside the Vite development server via stdio. The MCP server communicates with the browser over a WebSocket connection. Inside the browser, two modules handle the framework side: an MCP client (injected by the Vite plugin) captures console logs and screenshots, while the MCP runtime (part of IWSDK core) exposes ECS and scene graph inspection. Underneath both, IWER (the Immersive Web Emulation Runtime) emulates the XR headset and controllers, and its RemoteControlInterface translates MCP tool calls into device state changes. The agent’s workflow is: write code, reload the page, observe the result via screenshots and console logs, query the scene and ECS state, diagnose issues, and fix — all without human intervention.

MCP Server — The Bridge (`vite-plugin-iwer`)

An MCP (Model Context Protocol) server that runs alongside the Vite development server and exposes 32 tools to the AI agent over stdio. The server bridges between Claude Code and the running browser application via WebSocket.

Tool inventory (32 tools across 5 categories)

Session management (3 tools)

Tool	Description
`get_session_status`	Get XR session state and device status
`accept_session`	Accept an offered XR session (equivalent to clicking “Enter XR”)
`end_session`	End the current active XR session

Device control (11 tools)

Tool	Description
`get_transform`	Get position/orientation of headset or controllers
`set_transform`	Move headset/controller to a world position and orientation
`look_at`	Orient a device to face a specific world position
`animate_to`	Smoothly animate a device to a target pose over time
`set_input_mode`	Switch between controllers and hand tracking
`set_connected`	Connect or disconnect an input device
`get_select_value`	Read the current trigger/pinch value
`set_select_value`	Set trigger/pinch pressure (0–1)
`select`	Perform a full press-and-release action
`get_gamepad_state`	Read all button and axis values on a controller
`set_gamepad_state`	Set button presses, thumbstick position, and other gamepad values

Observation (4 tools)

Tool	Description
`capture_canvas`	Screenshot the rendered XR view as PNG/JPEG/WebP
`get_console_logs`	Retrieve browser console output with level/timestamp filtering
`get_device_state`	Full snapshot of all device transforms and input state
`set_device_state`	Restore a complete device state snapshot

Scene inspection (2 tools)

Tool	Description
`get_scene_hierarchy`	JSON tree of the Three.js scene graph (names, UUIDs, entity indices)
`get_object_transform`	Local and global transform of any scene object

ECS debugging (11 tools)

Tool	Description
`ecs_pause`	Freeze all ECS system updates (rendering continues)
`ecs_resume`	Resume ECS execution
`ecs_step`	Advance exactly N frames while paused
`ecs_query_entity`	Inspect all components and their data on an entity
`ecs_find_entities`	Search entities by component type
`ecs_list_systems`	List all registered systems with execution order and enabled state
`ecs_list_components`	List all registered component types
`ecs_toggle_system`	Enable/disable individual systems at runtime
`ecs_set_component`	Write component values on a live entity without restarting
`ecs_snapshot`	Capture full ECS state for comparison
`ecs_diff`	Compare two ECS snapshots to identify state changes

Utility (1 tool)

Tool	Description
`reload_page`	Force-refresh the browser page

Browser-side MCP client (injected via `Vite` plugin)

JavaScript modules automatically injected into the running application that enable the server-side tools to function:

WebSocket client — persistent connection between the browser and the MCP server, with auto-reconnect and message queuing

Console capture — intercepts console.log/warn/error/info/debug and buffers them for retrieval; captures uncaught exceptions and unhandled promise rejections

Screenshot capture — captures the WebGL canvas as a base64-encoded image on demand

MCP Runtime (IWSDK core)

A runtime module inside the IWSDK core package that exposes framework-specific debugging tools. Installed automatically during World.create() and made available to the MCP server via window.FRAMEWORK_MCP_RUNTIME.

Key capability: the ECS debug hook monkey-patches the world’s update() loop to intercept frame execution, enabling pause, step, snapshot, and diff operations without any application code changes.

IWER Remote Control Interface

A programmatic API added to the Immersive Web Emulation Runtime (IWER) that enables code-driven control of the emulated XR headset and controllers. This is the layer that translates MCP tool calls like set_transform and select into actual state changes in the WebXR emulation.

Features:

Action queue with frame-synchronized processing

Duration-based animations (smooth interpolation between poses)

Full gamepad state control (buttons, axes, touches)

Input mode switching (controllers ↔ hand tracking)

Device connect/disconnect simulation

Project scaffolding — zero-setup AI integration

Running npm create iwsdk now produces a project that is fully configured for AI-assisted development out of the box:

Pre-configured MCP servers:

iwer — the 32-tool MCP server described above

iwsdk-rag-local — a RAG (Retrieval-Augmented Generation) server with semantic search over the entire IWSDK codebase (3,337 indexed code chunks, 8 search tools)

Pre-configured Claude Code skills (6):

Skill	Purpose
`iwsdk-planner`	Architecture planning with IWSDK best practices and anti-patterns
`catalog-assets`	Browse and integrate 3D assets from the starter asset library
`iwsdk-ui-panel`	Build and iterate on spatial UI panels using UIKitML
`click-target`	Add pointer interaction to entities
`preview-model`	Load and preview 3D models in the scene
`xr-mode-test`	Validate XR session mode switching

Pre-configured permissions: All 32 MCP tools and all 6 skills are pre-allowed — the developer never sees a permission prompt for standard AI operations.

Project knowledge: A PROJECT_CLAUDE.md file is generated containing IWSDK best practices, critical anti-patterns (the #1 cause of bugs: missing LocomotionEnvironment when locomotion is enabled), VR performance constraints (11–14ms frame budget), and ECS coding conventions.

CLI enhancements

--name, --features, and other flags for scripted, non-interactive project creation (enables CI and automation workflows)

--bundle flag for fully offline project creation — pre-packages all npm dependencies as .tgz files so projects can be created in air-gapped or restricted network environments

Key differentiators

vs. Native XR development (Unity, Unreal)

Aspect	Native engines	IWSDK + AI tooling
Iteration cycle	30–90s compile, deploy APK	Save file → instant refresh
AI feedback loop	AI writes code, human tests manually	AI writes code, sees result, self-corrects
Runtime debugging	Attach debugger, add breakpoints	AI pauses ECS, inspects entities, patches values live
Barrier to entry	Requires XR expertise + engine knowledge	Describe what you want in natural language
Sharing	Build → sign → side-load APK	Deploy to web URL or ProtoHub

vs. Other AI coding assistants on the web

Aspect	Generic AI + web framework	IWSDK + AI tooling
3D scene understanding	None — AI sees only code	AI queries scene graph, entity transforms, component data
XR input simulation	None	AI controls headset, controllers, hand tracking programmatically
Framework knowledge	Generic / hallucinated	RAG over 3,337 real code chunks with semantic search
Debug capability	Console logs only	Pause simulation, step frames, diff ECS state, toggle systems

User journey

Step 1 — Scaffold. The developer runs npm create iwsdk. The project is created with AI tooling pre-configured: MCP servers, skills, permissions, and project knowledge are all in place.

Step 2 — Prompt. The developer opens the project in Claude Code and describes what they want: “Build me a Beat Saber-style VR game with neon blocks and a laser sword.”

Step 3 — Build loop. Claude writes the application code using IWSDK APIs (verified against real documentation via the RAG server). It reloads the page, accepts the XR session, captures a screenshot of the rendered scene, and queries the scene hierarchy to understand the object layout. It moves the headset to a test position, triggers a grab interaction, reads the console logs, finds a bug, fixes it, and repeats — autonomously.

Step 4 — Tune. The developer says “The blocks feel too fast.” Claude finds the block entities via ecs_find_entities, reads their velocity components, reduces the speed live with ecs_set_component, and asks the developer to confirm. Once confirmed, Claude updates the source code to match the tuned values.

Step 5 — Share. The developer deploys to ProtoHub and shares the link. The recipient opens it in VR.

Technical requirements

Requirement	Detail
Runtime	Node.js 18+
Package manager	npm, `pnpm`, or yarn
Browser	Chromium-based (Chrome, Edge) for WebXR emulation
AI agent	Claude Code CLI with MCP support
Headset (optional)	Meta Quest 2/3/3S/Pro for on-device testing
Dependencies	`@modelcontextprotocol/sdk`, `ws` (WebSocket)

Packages affected

Package	Change
`vite-plugin-iwer`	MCP server, browser-side client injection, WebSocket bridge
`@aspect/iwsdk-core`	MCP runtime, ECS debug tools, scene inspection tools
`immersive-web-emulation-runtime` (IWER)	RemoteControlInterface, programmatic device control API
`immersive-web-emulation-runtime/devui`	Programmatic control mode UI
`create-iwsdk`	Claude scaffolding, CLI flags, offline bundle mode