Design

Input mappings

Updated: Sep 14, 2026

Input mappings define the standard actions that hardware input modalities produce. They bridge the gap between raw hardware signals from hands, controllers, head, and voice and the interaction types that users experience (ray casting, touch, and grab).

🎓 Adapting a controller-first game?

The Adapting an Existing Game for Hands⁠ interactive course on Meta VR Academy covers translating each controller action into a hands interaction, without a one-for-one remap.

Mapping hierarchy

Targeting and selection are the two foundation mappings.

Targeting identifies where the user is pointing
Selection confirms intent-to-act (tab or click) on that target.

The remaining mappings extend selection by adding duration, movement, or multiplicity.

Mapping	Builds on	Description
Targeting	—	Where is the user pointing?
Selection	Targeting	Act on the target (a tap or click).
Select and hold	Selection	Selection sustained over time (for context menus, dragging, or initiating manipulation).
One-point manipulation	Selection	Selection sustained with movement (grab and move, rotate, or scale with one hand).
Two-point manipulation	One-point manipulation	Two simultaneous one-point manipulations (scale or rotate with both hands).
Scroll	Selection	Selection combined with directional movement to navigate content.

See Input hierarchy for how the system prioritizes modalities and manages fallbacks when a mapping is unavailable.

Targeting

Video 1: Targeting with controller and hand

Targeting refers to the process of pointing at a specific object or location in the virtual environment to interact with it.

Depending on the input method and proximity to the target, various visual aids can assist the targeting process, ensuring precise and intuitive interaction. For example, when interacting with distant objects, a virtual ray - similar to a laser pointer - extends from the controller or hand, enabling users to point and target with precision.

Input modality	Targeting method	Interaction type
Hands	Hand ray (indirect), finger tip (direct)	Ray casting, Touch
Controllers	Controller ray (indirect), controller tip (direct)	Ray casting, Touch
Head	Head ray (center of FOV)	Ray casting
Gaze	Eye tracking	Cone casting
Mouse	Cursor	—

Selection

Video 2: Selection with controller and hand

Selection indicates; that the user intends to take action on a targeted object or control.

The method of selection varies depending on the input modality. For instance, users can select by pressing the trigger button on a controller or by pinching their hand with hand-tracking input when interacting with objects at a distance. Additionally, similar to physical object interactions, users can directly poke the object to perform a selection action, providing a seamless and immersive experience.

Input modality	Selection method	Interaction type
Hands	Pinch (indirect), poke (direct), palm grab	Ray casting, Touch, Grab
Controllers	Trigger button, grip button	Ray casting, Touch, Grab
Head	Paired with hands or controller for selection	—
Voice	Spoken command	—
Gamepad	Button press	—

Selection-based mappings

The following mappings build on selection by adding duration, movement, or multiplicity.

Select and hold

Video 3: Select and hold for grabbing a UI panel

Select and hold interaction refers to a type of interaction where the user taps on an object or element and holds their finger or pointer on it for a short period. This interaction is often used to access additional information, options, or actions related to the tapped object. For example:

A user can tap on an object with their finger and hold it there to access a context menu or additional information.
A user can select and hold the trigger button on the controller to initiate and hold an interaction, such as dragging an object or highlighting text.
A user can manipulate a virtual object with a hand ray, which enables users to grab and move objects. By pinching and holding, users can maintain the grabbed state and move the object to their desired location.

Input modality	Select and hold method	Interaction type
Hands	Pinch and hold, finger press and hold	Ray casting, Touch, Grab
Controllers	Trigger hold, grip hold	Ray casting, Grab

One-point manipulation

Video 4: One-point manipulation example

Mirroring real-life interactions, users can grasp and manipulate objects with ease. Grabbing can be achieved by pinching with their hand or by using the trigger button on the controllers, which allows for a natural and intuitive way to interact with virtual objects. Depending on the interface type and implementation behaviors, one-point grab could be used for other actions such as rotation or scaling.

Input modality	Manipulation method	Interaction type
Hands	Pinch and drag, palm grab	Grab
Controllers	Trigger hold and move, grip hold and move	Grab

Two-point manipulation

Video 5: Two-point manipulation example

Two-point manipulation allows users to use both hands or controllers to interact with virtual objects or interfaces.

Scaling

By moving the two grabbed points closer together, the user can scale the object down. By moving them further apart, the user can scale the object up.

Rotation

By changing the distance of the two grabbed points from the user’s body, the user can rotate the object. For example, if a user moves one hand closer to their body while keeping the other hand still, the object will rotate around the axis defined by the two grabbed points.

Input modality	Two-point method	Interaction type
Hands	Both hands pinch or palm grab	Grab
Controllers	Both controllers grip or trigger	Grab

Scroll

Video 6: Scrolling UI with hand

Similar to scrolling on 2D screen devices, users can scroll 2D content or 3D objects using various inputs. For instance:

Hand tracking can employ a swipe gesture to scroll through content.
A controller’s trigger button to scroll by grabbing, moving, and releasing.
- The thumbstick on the controller can also be used to scroll.

Input modality	Scroll method	Interaction type
Hands	Swipe gesture, pinch and drag	Touch, Ray casting
Controllers	Thumbstick, trigger grab and drag	Ray casting
Mouse	Scroll wheel	—
Microgesture	Thumb swipe	Gesture

Next steps

Designing experiences

Explore more design guidelines and learn how to design great experiences for your app users:

Input hierarchy: Learn how the system prioritizes input modalities and fallback logic.
Multimodality: Understand how the system orchestrates switching between input modalities.
Ray casting: Explore indirect interaction through ray casting.
Touch: Explore direct interaction through touch.
Grab: Explore object manipulation through grab.
Accessibility: Learn how to make your app accessible to all users.