Develop

Look and Pinch for 2D apps

Updated: Sep 4, 2026

This page explains how Look and Pinch input works for 2D Android panel apps, including the privacy-preserving interaction model, gaze targeting concepts, and requirements for compatibility.

How Look and Pinch works

Look and Pinch is the default input method on devices that ship without controllers. Instead of a controller pointer, the system uses eye tracking to determine where the user is looking, and hand gestures (such as a pinch) to confirm selections. Look and Pinch replaces the traditional controller-based raycast with a gaze-based targeting system.

For 2D panel apps, Look and Pinch input operates at the system level. Your app receives standard touch events when the user pinches to select an element, similar to how a mobile app receives tap events. Your app does not need to implement eye tracking or hand tracking directly.

Gaze interaction flow showing how the system infers UI understanding from the app's view structure, identifies interactable elements, renders hover highlights, translates pinch gestures to touch events, and delivers standard touch input to the app

Privacy preservation

Due to the sensitive nature of eye tracking data, the system does not expose gaze data to 2D panel apps. Apps do not receive hover events derived from gaze input. Instead, the system retrieves UI layout information from your app and uses it for interaction targeting and hover effect rendering.

This means:

Your app does not receive hover events from gaze input.
The system renders hover effects on behalf of your app by overlaying visual highlights on interactable elements.
Your app only receives standard input events (such as touch down and touch up) when the user performs a selection gesture.

This approach follows industry standards for privacy-preserving gaze interaction. This is similar to how mobile and tablet apps work: a user’s finger hovering over a button does not trigger hover events in the app. The app only knows when the button is pressed.

Why this matters for your app: Any user flows or functionality that rely on hover state will not work with Look and Pinch. Replace hover-dependent interactions with click-based alternatives.

Gaze targeting challenges

Eye tracking presents unique targeting challenges compared to controller-based pointing:

Noisy input: Eyes make rapid movements called saccades when switching focus between elements, with brief pauses called fixations in between. This constant movement makes direct gaze-as-pointer unreliable for precise targeting.
Calibration variability: Eye tracking accuracy depends on calibration quality, which can degrade when the device is repositioned. Some users are also more difficult to track accurately than others.
No visible cursor: Displaying a gaze cursor is generally undesirable. If the cursor is not perfectly aligned with the user’s gaze direction, the user will chase the cursor, creating a frustrating feedback loop.

To address these challenges, the system applies several techniques:

Filtering: Eye tracking data is smoothed to reduce the impact of saccades.
Snapping: The gaze point magnetizes to nearby interactable elements, preventing the user from selecting empty space between targets.
Angular sizing: Touch targets are scaled so that a 48dp element corresponds to approximately 3 angular degrees, providing a comfortable minimum target size for gaze selection.

UI understanding

For snapping and hover rendering to work, the system must know where interactable elements are within your app. This is called UI understanding.

The system infers UI understanding from your app’s view hierarchy and accessibility information. For most apps, this works automatically without developer intervention. The system identifies:

Which elements are interactable (buttons, checkboxes, sliders, and other clickable elements)
The size, position, and shape of each interactable element

When the system cannot infer this information correctly, you can provide it explicitly using standard Android framework APIs. See Provide UI understanding for implementation details.

System hover effect rendering

Because your app does not receive hover events, the system renders hover effects on your behalf. The system uses UI understanding to identify interactable elements and overlays a hover highlight when the user’s gaze rests on an element.

This means the hover-to-selection transition is handled entirely by the system. When the user performs a select gesture (pinching their fingers), the system fades out the hover effect and sends a touch event to your app. Your app then displays its own selection state (pressed, focused, or other visual feedback).

For the transition between system hover and app selection to appear seamless, design your selection affordances to match the shape and style of the system hover effect. See Privacy-preserving hover design for design guidance.

Angular sizing

Touch target sizes for gaze input are measured in angular degrees rather than physical pixels or density-independent pixels. Angular size represents the portion of the user’s field of view that an element occupies, and depends on three factors:

The size of the element in dp
The size of the app window in the virtual environment
The distance between the panel and the user

The system handles angular sizing automatically. It scales your app window so that 48dp corresponds to approximately 3 angular degrees, which is the target minimum for comfortable gaze selection. This means if you follow standard Android touch target sizing guidelines (minimum 48dp), your app meets the angular sizing requirement without additional work.

Diagram showing a user's eye looking at an app panel, with a 48dp button subtending approximately 3 angular degrees at the viewing distance

Important: After scaling, the app window may be resized to fit within the device’s field of view. Your app must support a range of window sizes. See App configuration for details on declaring supported window sizes.

Requirements summary

To properly support inferred UI understanding, your app should use at least the following framework versions:

Framework	Minimum version
Jetpack Compose	1.10.0
React Native	0.85

To launch on devices with Look and Pinch as the default input, your 2D Android app must meet the following requirements:

Requirement	Description
No gaze permissions	Your app must not request eye tracking permissions outside of hybrid or immersive cases. See Make your existing app compatible with Meta VR devices.
Proper UI understanding	Your app must provide a properly constructed view hierarchy so the system can identify interactable elements. See Provide UI understanding.
Minimum target size	Touch targets must be at least 48dp. See Eyes Best Practices for sizing guidance.
No hover-dependent flows	Your app must not require hover state or use on-hover progressive disclosure, since apps do not receive hover events. See Eyes Best Practices.

Items not listed above are recommendations. Following the guidance in this documentation set improves the Look and Pinch experience but is not required for launch.

Provide UI understanding: Implementation details for each framework (Android Views, Jetpack Compose, React Native)
Privacy-preserving hover design: Design guidance for hover effects and selection transitions
App configuration: Manifest configuration and window sizing
Eyes Best Practices: UI layout, target sizing, color theming, and progressive disclosure guidance