Develop

Provide UI understanding for Look and Pinch

Updated: Sep 4, 2026

The Look and Pinch system must be able to identify interactable elements in your app for targeting and hover rendering. This process is called UI understanding. If the system cannot identify an element, users cannot interact with it using Look and Pinch.

Overview

For most apps, UI understanding works automatically. The system infers interactable elements from your app’s view hierarchy and accessibility information without any developer intervention.

There are cases where the system does not correctly infer an element’s state. In those cases, use the guidance on this page to identify and fix the issue. The three components of UI understanding are:

Importance for interaction: Whether an element is relevant to the gaze system, either as an interaction target or as a view that can occlude other interactable elements (such as menus or windows).
Interactability: Whether an element can be acted upon (clicked, toggled, scrolled).
Shape and bounds: The size, position, and shape of the element for targeting and hover rendering.

Mark elements as important for interaction

The system uses importance for interaction to determine which elements participate in occlusion culling. Elements that are not important are excluded from occlusion culling entirely. Only interactable elements (described in the next section) become gaze targets.

The system evaluates importance and interactability through heuristics and cross-checks them: if the system determines an element is certainly not important, it is also treated as not interactable. If the system cannot determine importance, it defaults to the element’s interactability state. In practice, an element is considered important if it is interactable, or if it is not interactable but may still occlude other interactable elements.

Jetpack Compose

For Jetpack Compose, importance for interaction is not configurable through a dedicated API. The system uses heuristics to determine which elements are important for interaction. Interactability (described in the next section) is the primary filter for determining which elements are gaze targets.

Android Views

For Android Views, an element is considered important for interaction if any of the following are true:

view.isClickable() returns true
- Set in code with setClickable(), or in XML with android:clickable
view.isLongClickable() returns true
- Set in code with setLongClickable(), or in XML with android:longClickable

Note

View.isClickable() indicates that the view is hittestable, not necessarily that it is interactable. A view that returns true for isClickable() participates in hit testing and can occlude other views behind it. This includes both interactable views (such as buttons) and non-interactable hittestable elements (such as menu backgrounds or dialog bodies).

React Native

React Native follows similar checks as Android Views, since React Native core components map directly onto Android Views. There is no additional override mechanism for importance at this time.

Mark elements as interactable

Elements marked as interactable become gaze targets, and the system’s targeting algorithms magnetize the gaze point to these targets.

Decision flow showing how the system determines if a UI element is interactable: first checks if the element is not important for interaction and excludes it, then checks hasOnClickListener, then checks Modifier.clickable or combinedClickable, and finally marks the element as not interactable if none of the checks pass

Jetpack Compose

For Jetpack Compose, interactable state is inferred from accessibility node information, which is derived from composable semantic properties⁠.

An element is considered interactable if any of the following are true:

Condition	How it is set
`info.isClickable()`	Composables with an `OnClick` semantic property. `Modifier.clickable` with `enabled=true` sets this automatically.
`info.isLongClickable()`	Composables with an `OnLongClick` semantic property. `Modifier.combinedClickable` with a non-null `onLongClick` parameter sets this automatically.
`info.isCheckable()`	Composables with a `ToggleableState` semantic property. `Modifier.toggleable` and `Modifier.triStateToggleable` set this automatically.

Example: Making a composable interactable with Modifier.clickable:

Box(
    modifier = Modifier
        .size(48.dp)
        .clickable(enabled = true) { onItemSelected() }
) {
    Text("Select")
}

Android Views

The system infers interactable state from each View in two steps.

Step 1: The element is considered not interactable if any of the following are true:

Condition	How it is set
`view.isEnabled()` returns `false`	`setEnabled()` in code, or `android:enabled` in XML
Element is not visible to the user	Determined by the system when `alpha` or `transitionAlpha` is zero, or `visibility` is not `VISIBLE`

Step 2: If the element passed Step 1, it is considered interactable if any of the following are true:

Condition	How it is set
`view.hasOnClickListeners()`	`setOnClickListener()` in code
`view.hasOnLongClickListeners()`	`setOnLongClickListener()` in code
`view.isContextClickable()`	`setContextClickable()` in code, or `android:contextClickable` in XML
View implements `Checkable`⁠	Checkbox, Switch, RadioButton, and similar widgets
View extends `AbsSeekBar`⁠	SeekBar, RatingBar, and similar slider widgets

React Native

React Native follows the same checks as Android Views, since React Native core components (such as View⁠) map directly onto Android Views through the UI manager.

Provide element shape and bounds

Once an element is identified as interactable, the system needs to know the element’s size and shape. This information is used for two purposes:

Targeting: The gaze magnetizes to the bounding box of the element.
Hover rendering: The system draws a hover overlay that matches the element’s shape.

Hierarchy diagram showing wrong approach with individual child click listeners causing separate highlights versus correct approach with parent card click listener causing a single card highlight

Jetpack Compose

In recent versions of Jetpack Compose UI, shapes are passed through accessibility extras automatically. The shape can be set on a composable through:

Modifier.graphicsLayer with clip = true (also set by Modifier.shadow with elevation > 0)
Modifier.border
Modifier.background

Example: Setting a shape on a button composable:

Button(
    onClick = { onAction() },
    modifier = Modifier.clip(RoundedCornerShape(12.dp)),
) {
    Text("Action")
}

If no shape is set, the system falls back to info.getBoundsInScreen(), which derives bounds from the composable’s layout bounds (relative to the Compose root) with adjustments from merging, clipping, and hit-testing.

Android Views

The system infers shape and bounds from the View’s outlineProvider. Each View can have a separate outline provider that defines its shape.

If the default outline does not match your element’s visual appearance, override the outline provider:

view.setOutlineProvider(new ViewOutlineProvider() {
    @Override
    public void getOutline(View view, Outline outline) {
        outline.setRoundRect(0, 0, view.getWidth(), view.getHeight(), 16f);
    }
});

For more complex shapes, use outline.setPath(path) with a custom Path object.

React Native

React Native follows the same approach as Android Views for shape and bounds inference. Core component props that affect shape (such as borderRadius) translate to the Android View’s outline provider through the UI manager.

If the system does not correctly infer your element’s outline, verify that the React Native component’s style properties correctly map to the underlying Android View’s outline.

Test UI understanding in Meta Spatial Simulator

Use Meta Spatial Simulator’s Look and Pinch testing mode to inspect your app without a physical device. Enable Show Interactive Elements for a complete visualization of the interaction structure that Meta Horizon OS detects. The overlay helps you find missing targets, unexpected targets, and incorrect bounds.

Meta Spatial Simulator outlining the interactable elements in a recipe app

Troubleshooting

Element does not respond to gaze selection

Symptom: An element that should be interactive (such as a button or checkbox) does not receive a hover highlight and cannot be selected using Look and Pinch.

Solution: Verify the element is marked as both important for interaction and interactable. For Android Views, check that the element has a click listener attached (setOnClickListener()) or is an instance of Checkable or AbsSeekBar. For Jetpack Compose, verify the composable includes Modifier.clickable or Modifier.toggleable. See the interactability tables above for the full list of conditions.

Hover highlight does not match the element’s visual shape

Symptom: The system hover effect renders as a rectangle around an element that has rounded corners, a circular shape, or a custom outline.

Solution: The system derives the hover shape from the element’s outline provider (Android Views) or shape modifier (Jetpack Compose). For Android Views, override the outlineProvider to define the correct shape. For Jetpack Compose, set the shape via Modifier.graphicsLayer, Modifier.border, or Modifier.background. See Provide element shape and bounds for code examples.

Look and Pinch for 2D apps: Overview of the Look and Pinch interaction model
Privacy-preserving hover design: Design guidance for hover effects and selection transitions
App configuration: Manifest configuration and window sizing
Eyes Best Practices: Target sizing, layout, and design best practices
Android accessibility semantics⁠: Jetpack Compose semantic properties reference
ViewOutlineProvider⁠: Android View outline provider API reference