Transitioning to 3D development

Updated: Sep 17, 2025

This guide will help you transition from 2D development to 3D VR development with Spatial SDK. It provides an overview of concepts unique to VR and frequently used APIs.

The 3D environment

In VR, the immersive scene is the 3D space that the player can see and interact with while they’re wearing the headset. In Spatial SDK, you create and edit scenes using Spatial Editor. A scene can either completely hide the user’s real world surroudings, or use passthrough mode to keep their surroundings visible while overlaying virtual content onto it.

Responding to the user’s actions

In 2D development, the user interacts with your app through a physical screen. The screen detects and handles user input, and the app responds accordingly. Panels in 3D space fulfill the same purpose in VR, but there is an additional point of interaction: the motion of the user. Because the person wearing the headset can freely look and move around the immersive scene, you need to know where they’re looking, the location of their head, body, and hands in the world, and what action they’re performing. Spatial SDK has multiple APIs to provide that information. Tracking data is often provided as a pose, the combination of a transform (x,y,z) and a quaternion (w, x, y, z) for rotation.

Controller API: Provides tracking data about both physical controllers and hands.

AvatarBody API and AvatarAttachment: Provides body tracking data.

This code sample uses the Controller and AvatarBody APIs to detect if the user is pressing the menu button on their left controller.

  val leftController =
    Query.where { has(AvatarBody.id) }.filter { isLocal() and by(AvatarBody.isPlayerControlledData).isEqualTo(true) }
      .eval()
      .first()
      .getComponent<AvatarBody>()
      .leftHand
      .tryGetComponent<Controller>()

  if (leftController != null && leftController.getPressed(ButtonBits.ButtonMenu)) {
      onMenuButtonPressed()
  }


  if (leftController != null && leftController.getReleased(ButtonBits.ButtonMenu)) {
      Log.i("MrukInputSystem", "Menu button released")
  }

  /** returns true if the button was pressed this frame */
  fun Controller.getPressed(button: Int): Boolean {
  return (buttonState and changedButtons and button) != 0
  }

  /** returns true if the button was released this frame */
  fun Controller.getReleased(button: Int): Boolean {
      return buttonState and button == 0 && changedButtons and button != 0
  }

Scene API: Provides controller and head pose data as well as multiple methods to affect the overall scene.

This code sample uses the Scene API to adjust the environment and move the user to the world origin.

  class MyActivity : AppSystemActivity() {
      override fun onSceneReady() {
          super.onSceneReady()

          // Set the reference space for tracking
          scene.setReferenceSpace(ReferenceSpace.LOCAL_FLOOR)

          // Enable/disable passthrough for MR mode
          val isMrMode = scene.isSystemPassthroughEnabled()

          // Set view origin position
          scene.setViewOrigin(0f, 0.0f, 2.0f, 0.0f)

          // Update environment lighting
          scene.updateIBLEnvironment("chromatic.env")
      }
  }

Make your UIs repond to user movement

On a mobile device, your app’s UI is always easily available to the user. To use an app, they simply look at the device’s screen. However, an immersive scene is in 3D, and the user can move and look around it freely. Since your app’s UI appears as a flat panel fixed in the immersive scene, users might walk past it or ignore it. To fix this, you can track the user’s gaze and update the UI position to keep it within their field of view.

Getting head position

This function shows how to find and retrieve the user’s head position in VR space. The code searches for the head attachment component and returns its current 3D position and rotation as a Pose object.

Example: Showcases/focus/app/src/main/java/com/meta/theelectricfactory/focus/Utils.kt

// Function to get the pose of the user's head
fun getHeadPose(): Pose {
  val head =
      Query.where { has(AvatarAttachment.id) }
          .eval()
          .filter { it.isLocal() && it.getComponent<AvatarAttachment>().type == "head" }
          .first()
  return head.getComponent<Transform>().transform
}

Did you find this page helpful?