Build Your Own Synthetic Environment Server

1. Open the Unity project containing the synthetic environment you want to use for passthrough simulation (henceforth referred to as the “server project”). Note: the server project cannot be an XR project.
2. Import the Synthetic Environment Builder package using the package manager.
3. Under SynthEnvServer, open Scene Annotation Tool:
   
4. An editor window will pop up:
   
5. Click Initialize Scene.
6. Now your server project should be capable of passthrough simulation. To verify, run it in play mode. This will serve as the synthetic environment server. Meanwhile, run an MR application in Meta XR Simulator. You should see its passthrough content coming from the server project.

1. Select a game object in the scene.
   
2. In the Scene Annotation Tool, make sure the Selected Object is correct, and then click Make 3D Scene Entity:
   
3. The game object should now be surrounded by a bounding box:
   
4. Apply the correct semantic label. Adjust center/size if needed.
   

1. Select the game object where the 2D entity plane is going to be attached to, for example, a desk:
   
2. In the Scene Annotation Tool, make sure the Selected Object is correct, and then click Make 2D Scene Entity:
   
3. This will create a plane under the selected game object:
   
4. Adjust the position and size of the plane to be at the desired place. Make sure its orientation accords with the following rules:
   • For ceilings, floors, and walls, the +y must point into the room.
   • For walls, doors, and windows, +x must point right, +z must point up, and +y must point into the room.
   • For other panels, +y is the up direction.
5. Apply the correct semantic label.
   

1. Copy-pasting a bounded 2D entity plane is not currently supported.
2. Each scene must have exactly one ceiling and one floor. Aside from that, you can freely choose which objects to label according to your particular use case.
3. Scene entities are highlighted by Unity Scene Gizmos. You can choose to not have them highlighted by unchecking Bounded 2D/3D Entities from the scene gizmos menu:
   

In the case of multiplayer simulation, it is helpful to know where each player is located when the server is running. The Synthetic Environment Builder offers the ability to create “position marks” for each player. Each position mark is a uniquely colored cube in the scene:

1. Make sure the view of the main camera includes the entire scene. We recommend making it top-down.
2. Add a Unity layer named Position Marks:
   

At this point, your server project should be ready to go. To validate, run it in play mode and connect it to an MR application running in the Meta XR Simulator. Check whether the application has correct passthrough and scene information. We recommend using the Unity Scene Manager sample. Then, you can build your server project and use it as a synthetic environment server.