Develop

Build Your Own Test Room

Updated: Sep 3, 2026

Overview

The Synthetic Environment Builder lets you use your own synthetic environment for mixed reality simulation in the Meta XR Simulator. It is a UPM package that you can import into your Unity project containing a synthetic environment, and turn that project into a Synthetic Environment Server.

Prerequisites

  • A Unity project containing the synthetic environment you want to use for passthrough simulation. This page calls it the server project.
  • The com.unity.postprocessing package, version 3.2.2. The Builder declares it as a dependency, so Unity resolves it when you add the package.
  • A scene where each room has exactly one floor and exactly one ceiling. The Builder builds a separate room layout for every room in the scene.

Install the Builder

The Builder ships as a separate download rather than inside the Meta XR Simulator package. Download it from Meta XR Simulator - Synthetic Environment Builder, then add the com.meta.xr.simulator.synthenvbuilder package to the server project with Window > Package Manager > + > Add package from disk.

Getting started

  1. Open the server project with the Builder package installed.
  2. Under SynthEnvServer, open Scene Annotation Tool:
    Illustrating the Scene Annotation Tool menu option in the Unity Editor.
  3. An editor window will pop up:
    Illustrating the graphical interface of the Scene Annotation Tool editor window.
  4. Click Initialize Scene. The button becomes disabled once the server object exists in the scene, which is how you confirm this step worked.
  5. Run the server project in Play mode. It now acts as the synthetic environment server. Run an MR application in Meta XR Simulator and confirm that its passthrough content comes from the server project.

How the server connects

Initializing the scene adds a server component to it. Entering play mode starts a gRPC server on localhost:33792, which is the port Meta XR Simulator looks for a synthetic environment on. Leaving play mode stops it.
In the Unity Editor the server runs while Play mode is active: passthrough in the simulator lasts exactly as long as the server project is playing, and stops when you exit Play mode. If you build the server project instead, the same server starts and stops with that player process.
Only one process can hold port 33792, so a built-in room and a server project cannot serve at the same time. Exit Play mode to release the port cleanly. If the simulator reports that the environment failed to restart on that port, Force close in the Environments popover kills whatever process holds it. That is the Unity Editor when the Builder is running, so save your work first and reserve Force close for a process you cannot stop any other way.

Adding scene information

The next thing to do is annotate your synthetic environment with scene information.

To label a 3D entity

  1. Select a game object in the scene.
    A chair is selected in the Unity game editor.
  2. In the Scene Annotation Tool, make sure the Selected Object is correct, and then click Make 3D Scene Entity:
    Scene Annotation Tool UI after the chair is selected.
  3. The game object should now be surrounded by a bounding box:
    The chair is surrounded by a red wire box in the Unity Editor.
  4. Apply the correct semantic label. Adjust center/size if needed.
    Scene Annotation Tool UI with center/size and semantic label.

To label a 2D entity

  1. Select the game object where the 2D entity plane is going to be attached to, for example, a desk:
    A desk is selected in the Unity game editor.
  2. In the Scene Annotation Tool, make sure the Selected Object is correct, and then click Make 2D Scene Entity:
    Scene Annotation Tool UI after the desk is selected.
  3. This creates a plane under the selected game object:
    A plane has been created but is in the wrong location (on the floor).
  4. Set the position, size and orientation of the plane to match the surface it represents. The Builder guesses the initial rotation and never validates it, so check the result in the Scene view against the image below.
    A wall with right orientation setup.
    The plane is at the right location (on the table surface).
  5. Apply the correct semantic label.
    Scene Annotation Tool UI with semantic label.
Note:
  1. Each room needs exactly one ceiling and one floor. Aside from that, you can freely choose which objects to label according to your particular use case.
  2. The label dropdowns offer a fixed set. For a 2D entity: CEILING, COUCH, DOOR_FRAME, FLOOR, TABLE, WALL_ART, WALL_FACE and WINDOW_FRAME. For a 3D entity: BED, COUCH, LAMP, OTHER, PLANT, SCREEN, STORAGE and TABLE.
  3. For a single-room scene you can skip room assignment: annotations with no room fall into a built-in default room. For more than one room, select Create Room Entity for each, then assign every annotation to its room with the Room dropdown in the Inspector.
  4. Scene entities are highlighted by Unity Scene Gizmos. You can choose to not have them highlighted by unchecking Bounded 2D/3D Entities from the scene gizmos menu:
    Unity Scene Gizmos with Bounded 2D and 3D Entities highlighted.

Optional: Displaying client positions in the server project

For multiplayer simulation, the Synthetic Environment Builder can create “position marks” that show where each player is when the server is running. Each client that streams passthrough or hand state gets a color-coded cube, plus a mark for each of its hands:
Top-down view of a synthetic living room environment with three colored cubes.
To add position marks:
  1. Make sure the view of the main camera includes the entire scene. Use a top-down view. The marks are excluded from the capture cameras, so they never appear in simulated passthrough.
  2. Add a Unity layer named exactly Position Marks. The Builder looks the layer up by that name:
    Unity "Tags & Layers" Window with "Position Marks" added as a layer.
A client gets a mark once it starts streaming passthrough or hand state. A client that only reads scene data gets none.

Validate the server project

Run the server project in play mode and connect it to an MR application running in the Meta XR Simulator. Check whether the application has correct passthrough and scene information. Use the Unity Scene Manager sample.
Synthetic office scene with all scene entities highlighted in blue. Then, you can build your server project and use it as a synthetic environment server.

Next steps

Built-in rooms and rooms created with the Synthetic Environment Builder provide Passthrough, Scene, and Depth data. If you only need Scene data, use a lightweight JSON-based room instead. See Meta XR Simulator JSON Format Rooms.
Nav Logo
English (US)
© 2026 Meta