This system uses the HorizonOS InteractionNode API to extract UI elements from panels. It provides a provider API for other systems (such as ISDK) to consume the collected elements.

The system subscribes to PanelInteractionExtensionSystem as a PanelInteractionDelegate. When a panel is detected with a ViewRootImplInteractionExtension, the system installs a custom ViewRootImplInteractionExtension.Delegate that intercepts reportInteractionStructure callbacks.

When an InteractionStructure is reported, the system traverses its InteractionNodes and builds corresponding UiElement objects with world-space coordinates.

val panelSystem = PanelInteractionExtensionSystem(context)
val uiUnderstandingSystem = UiUnderstandingSystem(panelSystem)
// Access collected elements
val elements = uiUnderstandingSystem.getCollectedElements()
// Get elements for a specific panel
val panelElements = uiUnderstandingSystem.getElementsForPanel(panelEntityId)
// Resolve a specific element by panel + element ID
val element = uiUnderstandingSystem.getElementById(panelEntityId, elementId)

Signature

class UiUnderstandingSystem(panelInteractionExtensionSystem: PanelInteractionExtensionSystem) : SystemBase, PanelInteractionDelegate

Constructors

UiUnderstandingSystem ( panelInteractionExtensionSystem )

Signature

constructor(panelInteractionExtensionSystem: PanelInteractionExtensionSystem)

Parameters

panelInteractionExtensionSystem: PanelInteractionExtensionSystem The system that tracks panel windows and interaction extensions

Returns

UiUnderstandingSystem

Properties

executeCount : Long [Get]	The number of times this system's SystemBase.execute method has been called by the system manager. Incremented automatically before each execution cycle. Useful for performance monitoring. Signature var executeCount: Long
systemManager : SystemManager [Get][Set]	The SystemManager that this system is registered with. Provides access to other systems and the scene. Initialized during system registration. Signature lateinit var systemManager: SystemManager

Methods

associateSystemManager ( systemManager )	Associates this system with a SystemManager. Called during registration. Throws RuntimeException if the system is already associated — each system instance can only belong to one manager. Signature fun associateSystemManager(systemManager: SystemManager) Parameters `systemManager: SystemManager` The system manager that will own and execute this system.
delete ( entity )	Called when an SystemBase.delete is removed from the scene. Override to clean up any per-entity state this system maintains, such as cached components, tracking maps, or native resource handles. Signature open fun delete(entity: Entity) Parameters `entity: Entity`
destroy ()	Called during system shutdown to release all resources held by this system. Override to dispose of native handles, close streams, and clear any global state accumulated during the session. Signature open override fun destroy()
disable ()	Disables this system. When disabled: execute() does nothing No elements are collected Signature fun disable()
enable ()	Enables this system. When enabled, the system will process panels and collect elements. Signature fun enable()
equals ( other )	Checks if this system is equal to another object. Two systems are considered equal if they are the same instance or share the same Kotlin class, regardless of configuration state. Signature open operator override fun equals(other: Any?): Boolean Parameters `other: Any?` The object to compare with. Returns `Boolean` True if the other object is the same system instance or the same system class.
execute ()	Called each frame by the SystemManager to perform this system's per-tick logic. Override to query entities with relevant components and apply updates. Signature open override fun execute()
getCollectedElements ()	Returns a snapshot of the currently collected UI elements from all tracked panels. Signature fun getCollectedElements(): List<WorldSpaceUiElement> Returns `List` List of WorldSpaceUiElements representing interactable UI elements
getDependencies ()	Returns the execution ordering dependencies for this system. Override to declare which other systems must run before or after this one using SystemDependencies. Signature open fun getDependencies(): SystemDependencies? Returns `SystemDependencies?` The dependency configuration, or null if this system has no ordering constraints.
getElementById ( panelEntityId , elementId )	Resolves a specific element given a panel entity ID and element ID. Signature fun getElementById(panelEntityId: Long, elementId: Long): WorldSpaceUiElement? Parameters `panelEntityId: Long` The entity ID of the panel `elementId: Long` The InteractionNode ID of the element Returns `WorldSpaceUiElement?` The element with the given ID, or null if not found
getElementsForPanel ( panelEntityId )	Returns the elements for a specific panel entity. Signature fun getElementsForPanel(panelEntityId: Long): List<WorldSpaceUiElement>? Parameters `panelEntityId: Long` The entity ID of the panel Returns `List?` List of elements for the panel, or null if the panel is not tracked
getPanelEntityIds ()	Returns the set of all panel entity IDs that currently have interaction structures. Signature fun getPanelEntityIds(): Set<Long> Returns `Set` Set of panel entity IDs
getScene ()	Returns the Scene that this system operates within. Delegates to the associated SystemManager to retrieve the active scene instance. Signature fun getScene(): Scene Returns `Scene` The scene containing the entities and rendering state for this system.
hashCode ()	Returns the hash code of this system, based on its Kotlin class identity. Consistent with the SystemBase.equals implementation — systems of the same class produce the same hash code. Signature open override fun hashCode(): Int Returns `Int` The hash code derived from this system's Kotlin class.
isEnabled ()	Returns true if this system is currently enabled. Signature fun isEnabled(): Boolean Returns `Boolean`
isVisualizationEnabled ()	Returns true if debug visualization is enabled. Signature fun isVisualizationEnabled(): Boolean Returns `Boolean`
onPanelInteractionDetected ( entity , interactionExtension )	Called when a panel with a ViewRootImplInteractionExtension is detected. Signature open override fun onPanelInteractionDetected(entity: Entity, interactionExtension: Any) Parameters `entity: Entity` The panel entity `interactionExtension: Any` The ViewRootImplInteractionExtension for this panel
onPanelInteractionRemoved ( entity )	Called when a tracked panel/ViewRootImplInteractionExtension is no longer detected. This can happen when: The panel entity no longer exists The panel no longer has an interaction extension An activity-based panel is destroyed Signature open override fun onPanelInteractionRemoved(entity: Entity) Parameters `entity: Entity` The panel entity
setFocusProvider ( provider )	Sets a custom focus provider. Pass null to restore the default (gaze-based). Signature fun setFocusProvider(provider: PanelFocusProvider?) Parameters `provider: PanelFocusProvider?`
setScene ( scene )	Sets the scene reference for debug visualization. Signature fun setScene(scene: Scene) Parameters `scene: Scene`
setVisualizationEnabled ( enabled )	Enables or disables debug visualization (colored bounding boxes around elements). Signature fun setVisualizationEnabled(enabled: Boolean) Parameters `enabled: Boolean`