Develop

Chapter 11: Scene Understanding

Updated: Jul 22, 2026

IWSDK scene understanding detects real-world planes, meshes, and anchors. This chapter shows how an AR or VR app can use that geometry.

What You’ll Build

By the end of this chapter, you’ll be able to:

Set up scene understanding systems for plane and mesh detection.

Automatically detect flat surfaces like floors, walls, and ceilings.

Detect complex 3D geometry including furniture and room structure.

Create stable anchor points that persist across tracking loss.

Build semantic-aware interactions based on detected object types.

Place virtual content accurately on real-world surfaces.

Overview

The scene understanding system provides:

Plane Detection - Automatically detect flat surfaces like floors, walls, and ceilings.

Mesh Detection - Detect complex 3D geometry including furniture and room structure.

Anchoring - Create stable reference points that persist across tracking loss.

The system makes an ECS entity for each detected shape.

Semantic Understanding - Meshes include semantic labels such as table, chair, and wall.

Key Components

SceneUnderstandingSystem - Core system that manages WebXR scene understanding.

XRPlane - Component for detected flat surfaces.

XRMesh - Component for detected 3D geometry.

XRAnchor - Component for anchoring objects to stable positions.

Quick Start

Here’s a minimal example to enable scene understanding:

import {
  World,
  SceneUnderstandingSystem,
  XRPlane,
  XRMesh,
  XRAnchor,
  SessionMode,
} from '@iwsdk/core';

// React to detected planes
class PlaneProcessSystem extends createSystem({
  planeEntities: { required: [XRPlane] },
}) {
  init() {
    this.queries.planeEntities.subscribe('qualify', (entity) => {
      console.log('New plane detected!', entity.object3D?.position);
    });
  }

  update() {
    this.queries.planeEntities.entities.forEach((entity) => {
      console.log('Existing plane: ', entity.object3D?.position);
    });
  }
}

// Create world with AR session and required features
World.create(document.getElementById('scene-container'), {
  assets,
  xr: {
    sessionMode: SessionMode.ImmersiveAR,
    features: {
      planeDetection: true,
      meshDetection: true,
      anchors: true,
    },
  },
}).then((world) => {
  // Register the scene understanding system
  world
    .registerSystem(SceneUnderstandingSystem)
    .registerComponent(XRPlane)
    .registerComponent(XRMesh)
    .registerComponent(XRAnchor)
    .registerSystem(PlaneProcessSystem);
});

System Setup

Step 1: WebXR Session Configuration

Scene understanding requires specific WebXR features to be enabled:

World.create(container, {
  xr: {
    sessionMode: SessionMode.ImmersiveAR,
    features: {
      planeDetection: true, // Enable plane detection
      meshDetection: true, // Enable mesh detection
      anchors: true, // Enable anchor creation
    },
  },
});

Important: Not all WebXR implementations support all features. The system will log warnings for unsupported features.

Step 2: Register System and Components

world
  .registerSystem(SceneUnderstandingSystem)
  .registerComponent(XRPlane)
  .registerComponent(XRMesh)
  .registerComponent(XRAnchor);

Step 3: System Configuration

Configure the system with these options:

world.registerSystem(SceneUnderstandingSystem, {
  configData: {
    showWireFrame: true, // Show/hide wireframe visualization for detected planes and meshes
  },
});

Understanding the Components

`XRPlane`

Represents detected flat surfaces in the real world. Do not manually create - these are automatically managed by the system.

Plane Properties

Entity Object3D - Visual representation of the plane (wireframe box by default).

_plane - Internal WebXR plane object (read-only).

`XRMesh`

Represents detected 3D geometry in the real world. Do not manually create - these are automatically managed by the system.

class MeshProcessSystem extends createSystem({
  meshEntities: { required: [XRMesh] },
}) {
  init() {
    // Query for detected meshes
    this.queries.meshEntities.subscribe('qualify', (entity) => {
      const isBounded = entity.getValue(XRMesh, 'isBounded3D');
      const semanticLabel = entity.getValue(XRMesh, 'semanticLabel');

      if (isBounded) {
        console.log('Object detected:', semanticLabel);
        const dimensions = entity.getValue(XRMesh, 'dimensions');
        const min = entity.getValue(XRMesh, 'min');
        const max = entity.getValue(XRMesh, 'max');

        console.log('Object size:', dimensions);
        console.log('Bounding box:', min, max);
      } else {
        console.log('Global mesh detected (room structure)');
      }
    });
  }
}

Mesh Properties

isBounded3D - Whether this is a bounded object vs global scene mesh, read more about bounded 3D meshes and scene meshes.

semanticLabel - Semantic classification such as table, chair, or wall.

min - Minimum bounding box coordinates.

max - Maximum bounding box coordinates.

dimensions - Calculated dimensions [width, height, depth].

_mesh - Internal WebXR mesh object (read-only).

`XRAnchor`

Anchors objects to stable real-world positions. User creates this component to anchor entities.

// Create an anchored object
const hologram = world.createTransformEntity(hologramMesh);
hologram.addComponent(XRAnchor);

// The system will automatically:
// 1. Create a stable anchor at the current world position
// 2. Attach the object to the anchored reference frame
// 3. Maintain stable positioning across tracking loss

Anchor Properties

attached - Whether the entity has been attached to the anchor group (managed by system).

Plane Detection

Understanding Plane Detection

WebXR plane detection identifies flat surfaces in the real world:

Horizontal planes: Floors, tables, desks.

Vertical planes: Walls, doors, windows.

Arbitrary planes: Angled surfaces like ramps.

Working with Detected Planes

// React to new planes
class PlaneProcessSystem extends createSystem({
  planeEntities: { required: [XRPlane] },
}) {
  init() {
    this.queries.planeEntities.subscribe('qualify', (entity) => {
      const planeObject = entity.object3D;
      const position = planeObject.position;
      const rotation = planeObject.quaternion;

      console.log('Plane detected at:', position);

      // Place content on the plane
      const marker = createMarkerMesh();
      marker.position.copy(position);
      marker.position.y += 0.1; // Slightly above the plane
      scene.add(marker);
    });

    this.queries.planeEntities.subscribe('disqualify', (entity) => {
      console.log('Plane lost:', entity.object3D?.position);
      // Clean up any content placed on this plane
    });
  }
}

Plane-based Content Placement

// Place objects on detected floor planes
class PlaneProcessSystem extends createSystem({
  planeEntities: { required: [XRPlane] },
}) {
  init() {
    this.queries.planeEntities.subscribe('qualify', (entity) => {
      const plane = entity.getValue(XRPlane, '_plane');

      // Check if it's a horizontal plane (floor-like)
      if (plane.orientation === 'horizontal') {
        const planePosition = entity.object3D.position;

        // Place a virtual object on the floor
        const virtualObject = createVirtualObject();
        virtualObject.position.copy(planePosition);
        virtualObject.position.y += 0.5; // Above the floor
        scene.add(virtualObject);
      }
    });
  }
}

Mesh Detection

Understanding Mesh Detection

WebXR mesh detection provides detailed 3D geometry:

Bounded meshes: Individual objects with semantic labels.

Global mesh: Generic scene mesh objects.

Semantic classification: The label of the detected mesh object if it is a bounded 3D mesh. Available semantic label values are listed in the WebXR semantic labels registry⁠.

Working with Detected Meshes

class MeshProcessSystem extends createSystem({
  meshEntities: { required: [XRMesh] },
}) {
  init() {
    this.queries.meshEntities.subscribe('qualify', (entity) => {
      const isBounded = entity.getValue(XRMesh, 'isBounded3D');
      const semanticLabel = entity.getValue(XRMesh, 'semanticLabel');

      if (isBounded) {
        handleBoundedObject(entity, semanticLabel);
      } else {
        handleGlobalMesh(entity);
      }
    });
  }
}

function handleBoundedObject(entity, semanticLabel) {
  const dimensions = entity.getValue(XRMesh, 'dimensions');
  const position = entity.object3D.position;

  switch (semanticLabel) {
    case 'table':
      console.log(`Table detected: ${dimensions[0]}x${dimensions[2]}m`);
      break;

    case 'chair':
      console.log('Chair detected');
      // Maybe spawn a virtual cushion or highlight
      break;

    case 'wall':
      console.log('Wall detected');
      // Could place artwork or UI panels
      break;
  }
}

function handleGlobalMesh(entity) {
  console.log('Room structure detected');
  // Could use for occlusion, physics, or environmental effects
}

Semantic-based Interactions

// Example: Interactive table system
class TableInteractionSystem extends createSystem({
  tables: { required: [XRMesh], where: [eq(XRMesh, 'semanticLabel', 'table')] },
}) {
  update() {
    this.queries.tables.entities.forEach((entity) => {
      const dimensions = entity.getValue(XRMesh, 'dimensions');
      const position = entity.object3D.position;

      // Create interactive surface
      this.createTableInterface(position, dimensions);
    });
  }

  createTableInterface(position, dimensions) {
    // Create UI or interactive elements for the table
    const interface = createTableUI(dimensions);
    interface.position.copy(position);
    interface.position.y += 0.01; // Just above table surface
    this.scene.add(interface);
  }
}

Anchoring

Understanding Anchors

Anchors provide stable positioning that persists it’s world location that won’t be changed with recentering the view.

Creating Anchored Content

// Anchor existing object
const existingObject = world.createTransformEntity(mesh);
existingObject.object3D.position.set(0, 1, -2);
existingObject.addComponent(XRAnchor);

Visual Feedback

Wireframe Visualization

The system can draw detected geometry as a wireframe:

// Configure during registration
world.registerSystem(SceneUnderstandingSystem, {
  configData: { showWireFrame: false },
});

// Or enable/disable wireframes
world.getSystem(SceneUnderstandingSystem).config.showWireFrame.value = true;

Common Patterns

AR Furniture Placement

// Place virtual furniture on detected floor planes
class FurnitureSystem extends createSystem({
  planeEntities: { required: [XRPlane] },
}) {
  init() {
    this.queries.planeEntities.subscribe('qualify', (entity) => {
      const plane = entity.getValue(XRPlane, '_plane');

      if (plane.orientation === 'horizontal') {
        const planePosition = entity.object3D.position;

        // Create anchored furniture
        const chairMesh = // Create your chair mesh here
        const chair = world.createTransformEntity(chairMesh);
        chair.object3D.position.copy(planePosition);
        chair.object3D.position.y += 0.4; // Chair height
        chair.addComponent(XRAnchor);
      }
    });
  }
}

Interactive Surface Detection

// Create interactive UI on detected tables
class SurfaceUISystem extends createSystem({
  meshes: { required: [XRMesh] },
}) {
  update() {
    this.queries.meshes.entities.forEach((entity) => {
      const semanticLabel = entity.getValue(XRMesh, 'semanticLabel');
      const isBounded = entity.getValue(XRMesh, 'isBounded3D');

      if (isBounded && semanticLabel === 'table') {
        this.createTableUI(entity);
      }
    });
  }

  createTableUI(tableEntity) {
    const dimensions = tableEntity.getValue(XRMesh, 'dimensions');
    const position = tableEntity.object3D.position;

    // Create UI panel
  }
}

Scene-Aware Physics

// Use detected planes as physics collision surfaces
class PlanePhysicsSystem extends createSystem({
  planeEntities: { required: [XRPlane] },
}) {
  init() {
    this.queries.planeEntities.subscribe('qualify', (planeEntity) => {
      // Add physics collision to detected planes
      const planeMesh = planeEntity.object3D;

      planeEntity.addComponent(PhysicsShape, {
        shape: PhysicsShapeType.Box,
        dimensions: [
          planeMesh.geometry.parameters.width,
          planeMesh.geometry.parameters.height,
          planeMesh.geometry.parameters.depth,
        ],
      });

      planeEntity.addComponent(PhysicsBody, {
        state: PhysicsState.Static,
      });
    });
  }
}

Troubleshooting

Common Issues

Scene understanding not working:

Verify AR session with related features are properly requested.

If you are running on a Meta Horizon OS device, ensure that the room scan (environment setup) is finished.

Planes/meshes not detected:

Ensure the environment has clear, well-lit surfaces.

Try manual room setup if available on the device.

Anchors not staying stable:

Verify ‘anchor’ feature is enabled in WebXR session.

Ensure adequate visual features for tracking.

Some devices have limitations on anchor count.

Performance issues:

Limit the number of entities with scene understanding components.

Disable wireframes when you do not need them.

Debug Tips

Log detection events:

class PlaneSystem extends createSystem({
  planeEntities: { required: [XRPlane] },
}) {
  init() {
    this.queries.planeEntities.subscribe('qualify', (entity) => {
      console.log('Plane detected:', entity.getValue(XRPlane, '_plane'));
    });
    this.queries.planeEntities.subscribe('disqualify', (entity) => {
      console.log('Plane lost');
    });
  }
}

Visualize bounding boxes:

class MeshSystem extends createSystem({
  meshEntities: { required: [XRMesh] },
}) {
  init() {
    this.queries.meshEntities.subscribe('qualify', (entity) => {
      const isBounded = entity.getValue(XRMesh, 'isBounded3D');
      if (isBounded) {
        const min = entity.getValue(XRMesh, 'min');
        const max = entity.getValue(XRMesh, 'max');

        // Create debug bounding box visualization
        const box = createBoundingBoxHelper(min, max);
      }
    });
  }
}