Hand Gesture
In this tutorial, you’ll learn how to use Hand Tracker to create customized hand tracking effects. You’ll also learn how to show different objects with the event and rotate objects using the Visual Scripting panel.
This article covers the following:
- Adding Hand Tracker
- Adding 2D resources
- Controlling rotation with Visual scripting
- Controlling objects’ visibility with hand gesture
Prepare
Start by downloading the HandGesture package and locate the Unfinished_HandGesture project.
You will see all the required assets have been imported into the Assets panel.
Open the drop-down in the Preview panel and choose + Add media. Select the hand.mp4 file.
Index Finger Gesture Controller
Firstly, let’s try to create an effect of a thunder sign attached to the index finger. We’ll make it only show up when it detects the index finger gesture.
Create an Index Finger Hand Tracker
We start by adding a Hand Tracker entity. We can do this by clicking + Add Object > AR tracking > Hand Tracker. A Hand Tracker will then be added to the Hierarchy. Create an Empty object and nest the Hand Tracker under it. Create another Empty object and nest it under the Hand Tracker.
Rename these object like below or other names that are reasonable for your hand gesture controller (you don’t need to add [MODEIFY ME] suffix):
Select the IndexF object, or the object with hand tracker, and find the Hand Tracker component in the Inspector panel. Change the Anchor from Hand Center to Index Finger tip. Then check Rotate AxisZ so that the object will have the same Z rotation as the finger.
Adjust Model, Material and Transform
Select the Lightning object in the Hierarchy panel.
In the Inspector panel, click + Add component > 3D > Model to add a Model component.
In the Model component, change Mesh to lightingBolt and Material to be the lightning_mat.
In the Transform component, change the Scale or X, Y, and Z to 0.1, and the Y Position to 2. This will create some distance between fingertip and the lightning object.
You will see the preview like this:
Control 3D lightning rotation with Visual Scripting
Next you need to control the rotation of the light object and keep it rotating all the time. Select the Lightning object to open the Inspector panel.
Find the Rotation parameter under Transform and click + to select Set Rotation. You can see the node was created in Visual scripting panel.
Next you need to control the rotate speed. To do this, you can go to the Visual Scripting panel, create a variable by clicking the Variables> Add variable, then change the variable name to RotateSpeed, change the Type to Number, change the Value to 240 (240 degree per second).
Next click the ‘plus’ button beside the variable name and select Get Variable. This will create a variable node in the visual scripting panel.
Next you need add the Get time node to control the rotation by clicking Add node > Time > Get time. In the same way, you can add Mod and Multiply node.
You also need a Combine node to convert these variables to Vector 3f. Add the Combine node and then click the data type on the Combine node and set the output data to Vector3f.
Now, let’s link the nodes together. Link trigger from the Update node to Set of Lightning so it will be updated every frame. Link the following together: Get time, Value1 of Multiply, RotateSpeed, Value2 of Multiply. Then link the value of Multiply to Value1 of Mod, to Rotation ofLightning. Change Value2 of Mod to 360, so that the Y angle will change between 0 and 360 (so it is smooth).
Feel free to change the RotateSpeed variable to control the speed of the rotation.
Control the Controller’s visibility
Next you need to hide and show the controller object using the Hand gesture Event. You can turn off the visibility of the Index Finger Gesture Controller to hide them at the beginning.
Let’s add the Gesture(Index Finger up) node by right clicking the panel and selecting it from the Hand category and add the Set visibility node from the Utility category. Duplicate the Set visibility node and check the Value. You’ll then need to drag the Index Finger Gesture Controller to the visual scripting panel and connect them together like below.
Hand Open Gesture Controller
For this session, we will try to implement the effect of having five accessories assigned to each fingers and they won’t show up until it detects the hand open gesture.
Create Five Finger Trackers
Just like ‘Creating an Index Finger Hand Tracker‘, you need to create another five finger trackers, but instead of adding the Lightning object, you need to add 3D Planes under each finger tracker.
You can start adding a Hand Tracker entity by clicking + Add Object > AR tracking > Hand Tracker. Then create an Empty object and drag the Hand Tracker under it.
In the same way, you can add a 3D plane through + Add Object > 3D > Plane and nest it under the Hand Tracker. Then you can change the name of each object to make it more sense (you don’t need to add the “MODIFY ME” suffix).
Go to the Hierarchy panel and select IndexF. Find the Hand Tracker component in the Inspector panel. Change the Anchor from the default Hand Center to Index Finger tip. Check the box for Rotate AxisZ.
You will now see a default 3D plane following your index finger in the Preview panel.
Adjust Material and Transform
Now select the Plane object in the Hierarchy panel. Change the material from default to watermelon_matin the Inspector panel under the Model component.
In the Transform component, change the scale (x, y, z) all to 0.258, and Position Y to 2. This will create some distance between fingertip and the plane.
Add other Hand Trackers
You can simply duplicate the index finger hand tracker 4 times, rename them and change the Anchor in each Hand Tracker component to Thumb tip/ Middle finger tip/ Ring finger tip/ Pinkie tip.
Then change the material to cactus_mat/ doughnut_mat/ ribbon_mat/ heart_mat. After these steps you will see the preview window below.
Control Controller’s visibility
Next you need to hide and show the controller object using the Hand gesture Event. You can turn off the visibility of the Index Finger Gesture Controller to hide them at the beginning.
Let’s add the (Hand Open) node by right clicking the panel and selecting it from the Hand category and add the Set visibility node from the Utility category. Duplicate the Set visibility node and check the Value. You’ll then need to drag the Hand Open Gesture Controller to the visual scripting panel and connect them together like below.
Refresh the preview and now we can see your very own effect in the Scene and Preview!
The logic will be: when you show index finger up gesture, it will the 3D lighting object. When you show the hand open gesture, it will show the other 2D objects and hide the lighting object.