Hand Grab Interactions
Updated: Aug 7, 2024
Design Guideline: Providing a comfortable hand tracking experience is essential for creating immersive and enjoyable apps. Refer to the
Hand Tracking Design Guide to learn about best practices and to minimize risks of user discomfort.
What are Hand Grab Interactions?
A
Hand Grab interaction provides a physics-less means of grabbing objects that’s specifically designed for hands and controller driven hands. For the controller version of grab, see
Grab Interactions. Unlike a
Grab interaction, a
Hand Grab interaction uses per-finger information to inform when a selection should begin or end using
HandGrabAPI.
In addition, the Hand Grab interaction enables hands to conform to a set of pre-authored poses during a selection. To learn about best practices when designing for hands, see
Designing for Hands.
By the end of this guide, you should be able to:
- Explain what a hand grab interaction is
- Define the essential properties and functionality of the HandGrabInteractor component.
- Define the essential properties and functionality of the HandGrabInteractable component.
- Explain how to specify what types of grabs the interaction supports.
- Describe how to configure the conditions that begin and end a hand grab interaction.
Note: If you are just getting started with this Meta XR feature, we recommend that you use
Building Blocks, a Unity extension for Meta XR SDKs, to quickly add features to your project.
How do Hand Grab Interactions Work?
The Hand Grab interaction has two components,
Hand Grab Interactor, which is attached to each hand via the
HandGrabInteractor prefab, and
Hand Grab Interactable, which is attached to each grabbable object. These two components communicate to determine whether a Hand Grab interaction has taken place and how the interaction should behave.
The
Hand Grab Interactor component is attached to your hands via the
HandGrabInteractor prefab.
HandGrabInteractor searches for the best
HandGrabInteractable candidate and visually aligns the synthetic hand with the interactable by using the
Hand Alignment property of
HandGrabInteractable and implementing
IHandGrabState.
Some key components, such as the
HandGrabAPI and SupportedGrabTypes, are discussed in the following sections, but the optional GripPoints and PinchPoints should be noted here:
- The GripPoint specifies an offset from the Wrist that can be used not only to search for the best HandGrabInteractable available, but also as a palm grab without a HandPose, and as anchor for attaching the object.
- The PinchPoint specifies a moving point at the center of the tips of the currently pinching fingers. It is used to align interactables that don’t have a HandPose to the center of the pinch.
When the target interactables have a HandPose, the Wrist is used for searching and aligning the item.
It is also possible to enforce a grab or release (selection/unselection) of a HandGrabInteractable by calling the method ForceSelect/ForceRelease.
The
HandGrabInteractable indicates not only whether an object can be grabbed, but how it can move with the interactor, which fingers should start and end the grab, and how the hand should align to the object.
The
HandGrabInteractable can also specify
HandGrabPoses,defining the handedness, visual hand pose, visual finger constraints, and surface to which a hand can align. If no HandGrabPose is provided, the grab will be a pose-less grab anchored at the interactable pose.
A grabbable object can be composed of multiple HandGrabInteractables.
At the
HandGrabInteractable, use the Scoring Modifier parameter to specify the strategy for finding the best grabbing point.
Position Rotation Weight: When using GrabSurfaces, this parameter indicates whether target poses that are closer to the actual hand should score better (no matter the rotation), or if poses with a rotation most similar to the rotation of the user’s hand should score better.
Note
For small to mid-sized objects, poses with a rotation most similar to the rotation of the user’s hand should score better, since it reduces the wrist motion.
Supported grab type determines which grab types the object responds to. Possible values are None, Everything, Pinch, Palm, and All.
Grab rules determine which fingers can start and end grabs. There are two sets of rules,
Pinch Grab Rules for pinch grabs, and
Palm Grab Rules for palm grabs. These rulesets apply so long as the
Supported Grab Type property includes them. Both rulesets have the following properties.
| Property | Description |
|---|
Thumb, Index, Middle, Ring, Max (pinky) | Determines if the finger is needed to trigger a grab. Can be Required, Optional, or Ignored. All fingers marked as Required must be performing the gesture in order to trigger a grab. If no finger is marked as Required, at least one finger marked as Optional must be performing the gesture to trigger the grab. Fingers marked as Ignored are completely ignored. |
Unselect Mode | Determines when the grab ends. Can be All Released or Any Released. In All Released mode, the grab ends when all required and optional fingers are released. In Any Released mode, the grab ends when at least one of the required fingers is released. If there are no required fingers, the grab ends when at least one of the optional fingers is released. |
Pinch Grab Rules Example
The key in the
HandGrabExamples scene can be pinch grabbed. All fingers are marked as
Ignored except for
Thumb and
Index, which are set to
Optional, so your thumb and index finger can trigger the grab.
Unselect Mode is set to
Any Released, so the grab will end when you release at least your thumb or index finger.
Triggering the pinch grab with the thumb and index finger, and then ending the grab by releasing the index finger.Palm Grab Rules Example
The torch in the
HandGrabExamples scene can be palm grabbed. All the fingers are set to
Optional, so you can trigger the grab using at least one finger or your thumb.
Unselect Mode is set to
All Released, so the grab will end when you release all of the fingers you used to trigger the grab.
Triggering the palm grab with just one finger or thumb, triggering the palm grab with all digits, and then ending the grab by releasing all of the digits.By default, when grabbing the interactable, it will align itself with the interactor and then move 1:1 with it each frame. To customize this behavior, attach an optional IMovementProvider component to the Interactable using the Add Component button. There are multiple components to choose from:
- MoveTowardsTargetProvider: This is the default behavior, but adding the component manually lets you adjust the alignment motion curve and timings:
- Travel Speed: The speed in m/s at which the object will move.
- Use Fixed Travel Time: Instead of speed in m/s, set the TravelSpeed to a fixed time in seconds.
- Travel Curve: The animation curve for how the speed will be applied to the object. By default, this applies some easing.
- MoveFromTargetProvider: Does not attract the object toward the interactor. Instead it anchors it at the interactor pose. Useful for constrained interactables.
- FollowTargetProvider: Continuously moves the interactor toward the interactable with some damping.
It is also possible to indicate how the visual hand will move in relation to the interactable and when a HandPose will be applied.
- AlignOnGrab: This behavior will automatically snap the hand visually to the interactable when a grab starts. This is the default behavior.
- AttractOnHover: This will progressively apply the final HandPose as the strength of a hover intensifies.
- Align Fingers On Hover: This will progressively apply the final poses for the fingers (not the wrist) as the user starts to close the hand during a hover.
- None: No hand pose is applied to the visual hand.
HandGrabInteractable can also specify which fingers are required to start and end a grab.
For example, a grab can be triggered when the middle or the index finger is pinching with the thumb, or when the ring and middle fingers are curled into a Palm grab. Releasing might require any or all of these fingers to release.
HandGrabPoses are the visual workforce behind HandGrab interactions. They provide the position and visual parameters for the hand.
The HandGrabPose specifies a pose relative to the object at which the wrist should anchor, additionally it can also complement this information with the following optional properties:
- HandPose: This option indicates that this HandGrabPose expects the hand to adopt a particular visual pose, and whether the fingers should be visually locked, constrained, or can move freely.
- Surface: This option defines a IGrabSurface along which snapping by this HandGrabPose can occur at any point, rather than just using the relative position of this pose for snapping.
When several HandGrabPoses are specified in a
HandGrabInteractable, the Interactable will interpolate between them based on the LocalScale and the Scale of the hand. With this approach, it is possible to generate HandGrabPoses with slightly different HandPoses at slightly different scales (such as 1x, 0.8x, and 1.2x), so that the hands of the users visually align properly with objects. Alternatively you can also choose to keep the scale of the users’ hands at a Fixed value using a FixedScaleHand.
An object can have several HandGrabInteractables with each indicating the different poses by which it can be grabbed via HandGrabPoses. To learn more about this workflow, check the cup in the HandGrabExamples sample scene.
We recommend that you always use HandGrabInteractables on objects with a scale of 1. If a different scale is required, scale a nested gameObject with the Mesh and Colliders but keep the root Rigidbody at scale 1.
The
HandGrabAPI is the core used to detect a Hand Grab Select or Unselect. It indicates when an IHand started and stopped grabbing, as well as the strength of the grabbing pose. To operate, the HandGrabAPI uses two IFingerAPIs implementations. By default,
FingerPinchGrabAPI is used for pinching grabs, and
FingerPalmGrabAPI is used for palm grabs. However, you can inject other APIs instead, such as the
FingerRawPinchAPI, which is useful when using controller driven hands.
Each
HandGrabInteractor can specify the
GrabTypes it will support. These can be Pinch, Palm, or both. Additionally, the
HandGrabInteractable can specify the
GrabTypes that it requires to perform a Grab (note that a Pinch-only interactable can not be grabbed by a Palm-only Interactor). These are used in conjunction with the
Grabbing Rules in the interactable.
Use with controller driven hands It is possible to use the
HandGrabAPI as-is when using Controllers-as-hands, the curl values being measured will be read directly from the animated hand. But it is recommended to use a
FingerRawPinchInjector component alongside the HandGrabAPI, this will force it to read the values directly from the controller-fed Pinch values instead, resulting in a more reliable interaction.
To enable visual Hand Grab Posing with Hand Grab Interactions, two additional components are required:
- A SyntheticHand overrides the wrist and joint hand data based on the hand grab interaction.
- A
HandGrabStateVisual component constrains finger joint rotations so they look as if they are perfectly wrapped around an object grabbed during a HandGrab interaction. This component can also override the wrist pose of the hand that’s attracted to any HandGrabInteractable that is marked with the Hand Alignment AttractOnHover.
An example of Hand Grab Posing can be seen in the HandGrabExamples scene. In this scene, there are separate SyntheticHands for Hands and controller driven hands. You can avoid this duplication and use one Hand Visual for both input device types if desired.
