Sluggo the Virtual Puppet: Part 4

Continued from Part 3.

Winter 2025 Review: Sluggo the Virtual Puppet

Sluggo says "hello"

This is a continuing journal of my project to design and build "Sluggo," a physical hand puppet that I digitized, rigged, and developed with hand gesture controls in Unity.

Bringing Sluggo to Life

The model was rigged with a skeleton and the skin was bound with weight paint. The final model was ready to bring into Unity for hand tracking control.

Step 1: Import to Unity

Sluggo in Unity

I exported the model from Maya to fbx format and imported it to Unity 2022.3.37f1. I used the same project that I had done some Mediapipe experimentation in earlier and already had many of the hand tracking control elements elements ready to go.

I had a moment of panic when I tested Sluggo's skeleton and found none of the joints would move any part of his body. Returning to Maya, I found that the "Animation" option was turned off in my export settings, which included skin weights. I re-exported with the option turned on and the mesh operated flawlessly in Unity.

Step 2: Test the Limits

Sluggo out of his "default" pose

I rotated each of Sluggo's joints that I wanted to control to determine the limits of motion and axis of rotation for each. For example, the lower lip joint could move about 75 degrees to close his mouth and each of the spine joints can bend about 30 degrees so that Sluggo can comfortably face the camera or look down.

Step 3: Hand Tracking

Courtesy of Mediapipe

I used the Unity plugin for Mediapipe for hand tracking. This system tracks one or two hands from a visual input like a webcam. The hand is tracked as a list of "landmarks," each representing a specific position on the hand in the 2D visual input. The developer can take those positions and use some math to determine the angle and gesture of each individual finger. For example, I can take the vector between landmarks 17 and 20 to determine the angle of the pinky finger. Once I know that, I can use that data to control the rotation of one of the eyestalk joints.

Step 4: Visual Scripting

My visual script to control Sluggo

I also took this project as an opportunity to try Unity's relatively new visual scripting platform. I am well acquainted with using C# with Unity and I've used Unreal's Kismet and Blueprint visual scripting, but I hadn't tried this one yet.

On initialization, the script tracks and stores the default rotation values of any joints that are controlled by hand tracking (this is all tucked away into a subgraph triggered by On Start). On every frame, the script analyzes the positions of certain hand landmarks (the lower right hand part of the graph) and sets the "bend" value for certain joints accordingly. Also on every frame, the script adjusts the rotation values of the controlled joints according to its current "bend" value (the unlabeled section just above the center of the graph). Each joint processes a subgraph I labeled "MySetRotation" that takes in the game object reference, the default rotation vector, the maximum rotation in degrees, and the bend value as arguments.

Mediapipe also generates a square to track where the hand is in the image, how large it is on the image, and what rotation it is on the image plane. I use this information to set Sluggo's position, rotation (perpendicular to the image plane), and proximity to the game camera. I also track the vector between landmarks 17 and 5 to how much the wrist is rotated and use that data to pivot Sluggo left or right.

Step 5: Completion