Hand Gameplay Showcase sample overview

Updated: May 11, 2026

Overview

This sample provides a collection of reusable hand tracking gameplay mechanics built for Meta Quest in Unreal Engine. It demonstrates teleportation, grabbing with pose override, throwing, button pushing, punching, two-handed aiming, and hand pose and gesture recognition through two custom plugins.

View the sample source code on GitHub⁠.

What you will learn

Implement teleportation, grabbing, throwing, and punching using hand tracking input

Use hand pose recognition with bone rotation matching across 18 bones

Build gesture recognition sequences from multi-pose state machines

Apply throwing velocity confidence with multi-tier scoring

Prevent fingertip pass-through using continuous overlap sphere collision

Stabilize tracking during loss events with UHandTrackingFilterComponent

Requirements

Meta Quest 2, Quest 3, or Quest 3S

Unreal Engine configured for Meta Quest development

OpenXR with Meta vendor extensions enabled

For setup instructions, see the Meta Quest Developer Hub documentation.

Get started

Clone the repository from GitHub, open the project in Unreal Engine, and build for Android. The project includes two custom plugins: OculusHandTools (with 5 C++ modules) and OculusUtils. For detailed build and configuration steps, see the project README⁠.

Explore the sample

Feature	What it demonstrates	Key concepts
Teleportation	Hand-tracked locomotion via pointing and confirming	Pose-based activation, arc preview
Grabbing with pose override	Grasping objects with custom hand poses	Grab detection, pose blending
Throwing	Physics-based throwing with velocity estimation	UThrowingComponent, multi-tier velocity confidence
Button pushing	Near-field interaction with pressable surfaces	UContinuousOverlapSphereComponent, fingertip collision
Punching	Impact detection using hand velocity	Velocity thresholds, collision events
Two-handed aiming	Bimanual aim mechanics	Dual-hand pose coordination
Hand pose recognition	Static pose matching using bone rotations	UHandPoseRecognizer, 18-bone matching, confidence scoring
Gesture recognition	Multi-pose sequence detection	UHandGestureRecognizer, state machine transitions

Runtime behavior

When running on a Meta Quest device, the sample presents interactive scenes where you use your hands to perform gameplay actions. The hand tracking system uses OpenXR with Meta vendor extensions to read bone rotations from 18 bones per hand. UHandPoseRecognizer encodes rotations using a compact string format with pitch, yaw, and roll values, then calculates a confidence score using the ErrorAtMaxConfidence threshold. UContinuousOverlapSphereComponent prevents fingertip pass-through during button pushing by maintaining collision detection every frame. UHandTrackingFilterComponent smooths hand position during brief tracking loss events.

Key concepts

Hand pose recognition

UHandPoseRecognizer matches the current hand skeleton against stored pose data by comparing bone rotations across 18 bones. Poses are encoded using a compact string representation that stores pitch, yaw, and roll per bone:

// Confidence is calculated based on rotation error distance
// ErrorAtMaxConfidence defines the threshold where confidence reaches 1.0
float Confidence = FMath::Clamp(1.0f - (Error / ErrorAtMaxConfidence), 0.0f, 1.0f);

Gesture recognition

UHandGestureRecognizer detects multi-pose sequences by running a state machine that tracks which poses have been recognized in order. Each gesture defines a sequence of required poses with timing constraints.

Throwing with velocity confidence

UThrowingComponent uses a multi-tier velocity confidence system that evaluates hand velocity samples over multiple frames. Higher confidence tiers require more consistent velocity direction and magnitude before releasing the thrown object.

Fingertip pass-through prevention

UContinuousOverlapSphereComponent maintains per-frame overlap checks at fingertip positions to prevent fast-moving fingers from passing through thin interactive surfaces like buttons.

Tracking loss stabilization

UHandTrackingFilterComponent detects when hand tracking is lost and holds the last known hand position with decay, preventing sudden jumps when tracking resumes.