Hands Best Practices

The following best practices guide the use of hands as inputs and interactions in VR app design and development. They aim to enhance user experience and ensure optimal performance.

This section examines the inherent limitations and challenges that impact performance and usability and offers strategies for mitigation through design or code improvements.

The headset’s sensors track within a specific area known as the tracking volume. Hands outside this area, such as resting by the thighs or placed too close to the device like on the cheek, will not be detected. The hand-tracking volume exceeds the display field of view, allowing for tracking even when hands are not visible to the user. The tracking volume varies depending on the hardware used.

Mitigation - Code: To compensate for hands moving outside the camera’s field of view, body tracking techniques are employed to infer the positions of the hands.

Mitigation - Design: To ensure that all relevant user motions are accurately tracked, it is recommended to design interactions that keep users’ movements within the tracking volume, avoiding the need for them to reach beyond it. For more guidance, refer to the comfort page.

Sensors play a vital role in tracking hand movements in virtual environments. They perform best when they have a clear view of the user’s hands. However, when parts of the hand are not visible to the sensors, the system attempts to estimate their positions. If significant portions of the hand are obscured, the virtual representation of the hand will disappear. The hand pose prediction model provides two key outputs: the estimated hand pose and the level of confidence in that prediction. If the confidence level falls below a certain threshold, the system will not display the hand. Obstacles that block the sensors’ view can include elements from the surrounding environment or the user’s own body, known as Self-Occlusions

Occlusions happen when hands are behind obstacles or interact with physical objects. Tracking accuracy can be affected by common obstructions such as long sleeves, hands in pockets, large rings, or black bandaids.

Self-Occlusions include scenarios where fingers are covered by other fingers, fingers point outward obscuring them from the headset-mounted display (HMD) cameras, or when hands overlap. These factors collectively influence the effectiveness of hand tracking technology.

Mitigation - Design: Aim to design interactions with minimal hand overlap; for example, having the index finger interact with a wrist button or palm menu on the other hand is a minimal overlap. The more one hand covers the other, the lower the accuracy. Additionally, interactions that can be performed with just one hand are always a good choice, as they not only enhance accessibility but also require less physical effort. This approach allows the hand tracking engine to effectively track both hands independently.

The quality and intensity of light can impact the system’s ability to accurately detect hand movements. Hand tracking has different lighting requirements than inside-out (head) tracking. In some situations, this could result in functional differences between head tracking and hand tracking, where one may work while the other has stopped functioning.

Mitigation - Design: Immersive experience headsets adjust to different light conditions. For optimal functionality, advise users to be aware of their environment’s lighting, perhaps by including a note when launching an application.

Rapid or erratic hand movements can challenge the system’s tracking accuracy. For example, when throwing a virtual object, the hand moves quickly, and the camera may not immediately capture the exact point of release. Nevertheless, the Move Fast app successfully demonstrates that playing fast-paced rhythm games using hand gestures is indeed possible.

Mitigation - Design: To improve tracking accuracy with fast-paced hand movements, consider these strategies:

By integrating these design elements, it is possible to significantly enhance the tracking performance of hand movements, particularly in fast-paced applications.

Tracking accuracy may be influenced by individual user traits. Factors such as long fingernails, specific types of nail polish, tattoos on the hands, accessories like rings, long sleeves, and diverse hand sizes and shapes can all impact performance.

Mitigation - Design: Enhance user interaction by offering a variety of input modalities and interaction choices. For instance, while long nails may hinder poke interactions, pinch actions using ray casting could be more effective. Alternatively, using a controller can serve as a reliable fallback option. For more information, please refer to the section on accessibility.

Below is a list of our recommended practices when using hands in an immersive experience:

Explore more design guidelines and learn how to design great experiences for your app:

For technical information, start from these development guidelines: