Use Simultaneous Hands and Controllers (Multimodal)

• Hand and controller gameplay: Use a controller in one hand as a tool or weapon (high accuracy, haptics, wide FoV), while keeping your other hand free to interact with objects or cast spells (high immersion).
• Single controller gameplay: If your experience only requires one controller for most interactions, allow the users to only pick up one controller and use their other hand for immersion and casual interactions when needed (for example, use a controller as a racket and use your free hand to pick up the ball and interact with menus).
• Use hands and controllers interchangeably for comfort: Use one hand for direct interactions with controls, while the other hand is holding a controller as a “remote control” for low effort indirect interactions.
• Instant transition between hands and controllers: With hand tracking active, we can instantly identify when the controllers are no longer in the hand. This minimizes the need for manual transitions and solves the problems of slow or failed auto transitions.
• Find my controllers: With controllers now tracked when using hands, the app can show the user where the controllers are when they want to pick them up. This allows smoother transition back to controllers without having to break immersion by turning on passthrough/ taking off the headset.

• The ‘in hand’ signal is based on various imperfect signals including hand and controller pose and controller signals. As a result, the system may indicate that the controller not in hand in certain scenarios where tracking is lost or inaccurate, or controllers are held still for some time. It is recommended to design with that limitation in mind (for example, avoid dropping objects from a hand due to false short transitions from controllers to hands).
• When the pause function is called, the application will switch back into the “non-simultaneous” mode that traditional hands+controllers apps run in, where the user can use either hands or controllers at a given time. The tracking system may take a few seconds to recognize and decide on the correct input to enable, depending on whether the user is holding controllers when this happens.
• When using Link on PC, pose data for controllers is unavailable when you’re not actively using them (such as when they’re lying on a table).

• Quest 2
• Quest Pro
• Quest 3
• Quest 3S

• Meta XR plugin v74 and above with OpenXR backend
• Unreal Engine 5.5 and above

• Multimodal input is incompatible with Inside-Out Body Tracking (IOBT) and Full Body Synthesis (FBS). You shouldn’t enable them together.
• Multimodal input cannot be enabled together with Fast Motion Mode (FMM). If both are enabled together, Multimodal will take precedence. As FMM is defined in the manifest, you may enable FMM at the app level, and then turn on multimodal only in specific experiences where FMM is less important.
• On Quest 2, Multimodal cannot be enabled together with LipSync
• Passthrough, Multimodal input, and Wide Motion Mode (WMM) cannot be enabled together. If they all are turned on together, the system will disable WMM.
• Full compatibility with capsense hands.
• Full compatibility with haptics.

1. In Unreal Editor, go to Project Settings → OculusXR → Mobile.
2. Set Hand Tracking Support to Controllers And Hands.

[/Script/OculusXRHMD.OculusXRHMDRuntimeSettings]
XrApi=NativeOpenXR
HandTrackingSupport=ControllersAndHands

Note: There is no Project Settings checkbox to enable multimodal directly. You enable multimodal entirely at runtime through C++ or Blueprints.

#include "OculusXRInputFunctionLibrary.h"

// Enable multimodal
bool bSuccess = UOculusXRInputFunctionLibrary::SetSimultaneousHandsAndControllersEnabled(true);

// Query whether multimodal is currently enabled
bool bEnabled = UOculusXRInputFunctionLibrary::IsSimultaneousHandsAndControllersEnabled();

• Set Simultaneous Hands And Controllers Enabled: Enables or disables multimodal. Returns true on success.
• Is Simultaneous Hands And Controllers Enabled: Returns the current multimodal state.

Important: Multimodal resets to disabled on every new session. Place the enable call in your BeginPlay event to ensure it activates each time the app starts.

#include "OculusXRInputFunctionLibrary.h"

// Set hand pose style when holding a controller
UOculusXRInputFunctionLibrary::SetControllerDrivenHandPoses(
    EOculusXRControllerDrivenHandPoseTypes::Natural);

// Query the current pose type
EOculusXRControllerDrivenHandPoseTypes CurrentType =
    UOculusXRInputFunctionLibrary::GetControllerDrivenHandPoses();

• Set Controller Driven Hand Poses: Sets the pose type.
• Get Controller Driven Hand Poses: Returns the current pose type.

#include "OculusXRInputFunctionLibrary.h"

bool bLeftHandTracking = UOculusXRInputFunctionLibrary::IsHandInteractionProfile(
    EOculusXRHandType::HandLeft);

Note: The OculusXR plugin does not expose a direct “is controller in hand” query. IsHandInteractionProfile indicates the active input source for a hand, not specifically whether the physical controller is in-hand versus detached.

• Hand tracking in Unreal
• OculusInputTest sample⁠
• Multimodal input (native)