Design

Input mappings

Updated: Mar 13, 2026

Input mappings define the standard actions that hardware input modalities produce. They bridge the gap between raw hardware signals from hands, controllers, head, and voice and the interaction types that users experience (ray casting, touch, and grab).

Mapping hierarchy

Targeting and selection are the two foundation mappings.

Targeting identifies where the user is pointing

Selection confirms intent-to-act (tab or click) on that target.

The remaining mappings extend selection by adding duration, movement, or multiplicity.

Mapping	Builds on	Description
Targeting	—	Where is the user pointing?
Selection	Targeting	Act on the target (a tap or click).
Select and hold	Selection	Selection sustained over time (for context menus, dragging, or initiating manipulation).
One-point manipulation	Selection	Selection sustained with movement (grab and move, rotate, or scale with one hand).
Two-point manipulation	One-point manipulation	Two simultaneous one-point manipulations (scale or rotate with both hands).
Scroll	Selection	Selection combined with directional movement to navigate content.

See Input hierarchy for how the system prioritizes modalities and manages fallbacks when a mapping is unavailable.

Targeting

Video 1: Targeting with controller and hand

Targeting refers to the process of pointing at a specific object or location in the virtual environment to interact with it.

Depending on the input method and proximity to the target, various visual aids can assist the targeting process, ensuring precise and intuitive interaction. For example, when interacting with distant objects, a virtual ray - similar to a laser pointer - extends from the controller or hand, enabling users to point and target with precision.

Input modality	Targeting method	Interaction type
Hands	Hand ray (indirect), finger tip (direct)	Ray casting, Touch
Controllers	Controller ray (indirect), controller tip (direct)	Ray casting, Touch
Head	Head ray (center of FOV)	Ray casting
Gaze	Eye tracking	Ray casting
Mouse	Cursor	—

Selection

Video 2: Selection with controller and hand

Selection indicates; that the user intends to take action on a targeted object or control.

The method of selection varies depending on the input modality. For instance, users can select by pressing the trigger button on a controller or by pinching their hand with hand-tracking input when interacting with objects at a distance. Additionally, similar to physical object interactions, users can directly poke the object to perform a selection action, providing a seamless and immersive experience.

Input modality	Selection method	Interaction type
Hands	Pinch (indirect), poke (direct), palm grab	Ray casting, Touch, Grab
Controllers	Trigger button, grip button	Ray casting, Touch, Grab
Head	Paired with hands or controller for selection	—
Voice	Spoken command	—
Gamepad	Button press	—

Selection-based mappings

The following mappings build on selection by adding duration, movement, or multiplicity.

Select and hold

Video 3: Select and hold for grabbing a UI panel

Select and hold interaction refers to a type of interaction where the user taps on an object or element and holds their finger or pointer on it for a short period. This interaction is often used to access additional information, options, or actions related to the tapped object. For example:

A user can tap on an object with their finger and hold it there to access a context menu or additional information.

A user can select and hold the trigger button on the controller to initiate and hold an interaction, such as dragging an object or highlighting text.

A user can manipulate a virtual object with a hand ray, which enables users to grab and move objects. By pinching and holding, users can maintain the grabbed state and move the object to their desired location.

Input modality	Select and hold method	Interaction type
Hands	Pinch and hold, finger press and hold	Ray casting, Touch, Grab
Controllers	Trigger hold, grip hold	Ray casting, Grab

One-point manipulation

Video 4: One-point manipulation example

Mirroring real-life interactions, users can grasp and manipulate objects with ease. Grabbing can be achieved by pinching with their hand or by using the trigger button on the controllers, which allows for a natural and intuitive way to interact with virtual objects. Depending on the interface type and implementation behaviors, one-point grab could be used for other actions such as rotation or scaling.

Input modality	Manipulation method	Interaction type
Hands	Pinch and drag, palm grab	Grab
Controllers	Trigger hold and move, grip hold and move	Grab

Two-point manipulation

Video 5: Two-point manipulation example

Two-point manipulation allows users to use both hands or controllers to interact with virtual objects or interfaces.

Scaling

By moving the two grabbed points closer together, the user can scale the object down. By moving them further apart, the user can scale the object up.

Rotation

By changing the distance of the two grabbed points from the user’s body, the user can rotate the object. For example, if a user moves one hand closer to their body while keeping the other hand still, the object will rotate around the axis defined by the two grabbed points.