Sluggo the Virtual Puppet: Part 4

Continued from Part 3.

Winter 2025 Review: Sluggo the Virtual Puppet

Sluggo says "hello"

This is a continuing journal of my project to design and build "Sluggo," a physical hand puppet that I digitized, rigged, and developed with hand gesture controls in Unity.

Bringing Sluggo to Life

The model was rigged with a skeleton and the skin was bound with weight paint. The final model was ready to bring into Unity for hand tracking control.

Step 1: Import to Unity

Sluggo in Unity

I exported the model from Maya to fbx format and imported it to Unity 2022.3.37f1. I used the same project that I had done some Mediapipe experimentation in earlier and already had many of the hand tracking control elements elements ready to go.

I had a moment of panic when I tested Sluggo's skeleton and found none of the joints would move any part of his body. Returning to Maya, I found that the "Animation" option was turned off in my export settings, which included skin weights. I re-exported with the option turned on and the mesh operated flawlessly in Unity.

Step 2: Test the Limits

Sluggo out of his "default" pose

I rotated each of Sluggo's joints that I wanted to control to determine the limits of motion and axis of rotation for each. For example, the lower lip joint could move about 75 degrees to close his mouth and each of the spine joints can bend about 30 degrees so that Sluggo can comfortably face the camera or look down.

Step 3: Hand Tracking

Courtesy of Mediapipe

I used the Unity plugin for Mediapipe for hand tracking. This system tracks one or two hands from a visual input like a webcam. The hand is tracked as a list of "landmarks," each representing a specific position on the hand in the 2D visual input. The developer can take those positions and use some math to determine the angle and gesture of each individual finger. For example, I can take the vector between landmarks 17 and 20 to determine the angle of the pinky finger. Once I know that, I can use that data to control the rotation of one of the eyestalk joints.

Step 4: Visual Scripting

My visual script to control Sluggo

I also took this project as an opportunity to try Unity's relatively new visual scripting platform. I am well acquainted with using C# with Unity and I've used Unreal's Kismet and Blueprint visual scripting, but I hadn't tried this one yet.

On initialization, the script tracks and stores the default rotation values of any joints that are controlled by hand tracking (this is all tucked away into a subgraph triggered by On Start). On every frame, the script analyzes the positions of certain hand landmarks (the lower right hand part of the graph) and sets the "bend" value for certain joints accordingly. Also on every frame, the script adjusts the rotation values of the controlled joints according to its current "bend" value (the unlabeled section just above the center of the graph). Each joint processes a subgraph I labeled "MySetRotation" that takes in the game object reference, the default rotation vector, the maximum rotation in degrees, and the bend value as arguments.

Mediapipe also generates a square to track where the hand is in the image, how large it is on the image, and what rotation it is on the image plane. I use this information to set Sluggo's position, rotation (perpendicular to the image plane), and proximity to the game camera. I also track the vector between landmarks 17 and 5 to how much the wrist is rotated and use that data to pivot Sluggo left or right.

Step 5: Completion

Sluggo the Virtual Puppet: Part 3

Continued from Part 2.

Winter 2025 Review: Sluggo the Virtual Puppet

Virtual Sluggo giving you the side eye

This is a continuing journal of my project to design and build "Sluggo," a physical hand puppet that I digitized, rigged, and developed with hand gesture controls in Unity.

Rigging Sluggo

The model was cleaned up and retopologized. Next, it needed a skeleton and skin binding so that the static mesh could be controlled dynamically through animation.

Step 1: Create the Skeleton

Sluggo in the X-ray machine

Most rigging tools are designed to create a skeleton for a humanoid bipedal character. Sluggo has a very different body from a human, so I needed to create a custom rig.

Starting at the root, Sluggo's has a few spine joints to the base of his head, which also serves as the pivot point for his jaw. One joint controls the lower lip, which should flap open and closed when talking, controlled by my thumb. The upper lip is controlled by two different joints, controlled by my middle and ring fingers, which should allow me to manipulate them individually to "scrunch up" his upper face. This effect is used to express frustration in Muppets like Kermit and Elmo.

The two joints from the top of the head from the bases of the eyestalks will remain static. The eyestalks are controlled by my index and pinky fingers.

Step 2: Paint the Weights

The influence of Sluggo's lower lip

Next, I painted the amount of influence each joint has on how the mesh deforms when it moves. Maya automatically set weight paints when the skin was bound to the skeleton, but these were not very good. I ended up flooding all influence weight to the root, then painstakingly hand-painted weights for each joint.

This section of the report only has two steps, but weight painting a deforming, cloth model took some time with tuning and refining to look good in different poses.

Next Step: Controlling in Unity

"Haven't I been through enough?"

Sluggo the Virtual Puppet: Part 2

Continued from Part 1.

Winter 2025 Review: Sluggo the Virtual Puppet

Sluggo in the digital realm

This is a continuing journal of my project to design and build "Sluggo," a physical hand puppet that I digitized, rigged, and developed with hand gesture controls in Unity.

Digitizing Sluggo

The first step in this multimedia project was to design and produce a physical hand puppet, affectionately named "Sluggo."

Step 1: Scanning

The scanning process

The physical puppet was set on a table atop a water bottle to hold it motionless during the scanning process. A hand scanner was used to digitize the puppet from every angle.

Sluggo the point cloud

The scanning process produced a cloud of colored points. The scan did a good job of capturing the details of the folds in the felt, the points where the pieces were sewn together, and the general color of the puppet. The point cloud needed to be converted to a triangular mesh with a material before further work could be performed.

Sluggo in Maya with 122K+ triangles

The 3D model is highly detailed with more than 122,000 triangles. I felt that performance would be improved in my planned real-time environment if I didn't have such a high-poly mesh to work with. Additionally, I could work more easily to edit the mesh if it was retopologized and quadrangulated (I can triangulate it for use in Unity later). I attempted to reduce the polygon count in Maya, but the command would either fail or produce an unsatisfactory result. 

Step 2: Correcting Errors in the Model with Meshmixer

Sluggo in Meshmixer

Autodesk Meshmixer is a relatively easy-to-use tool designed for use with 3D printing, but it can be repurposed for many types of 3D models. I sculpted the mesh to remove the few errors and aberrations from the scanning process, mostly around the base of the puppet (the end of the "sleeve") and to build up structure that was lost on the eyeballs. Unfortunately, this added a lot of polygonal detail in the areas where I was sculpting, mostly notably in the eyeballs.

Step 3: Retopologizing in Instant Meshes

Sluggo getting a new topology in Instant Meshes

Instant Meshes is a tool used for creating Instant Field-Aligned Meshes, based on a paper published by ACM. This tool helps create a topology that aligns well with the shape of a mesh, with the user able to paint direction lined directly ono the mesh, "combing" the lines of topology in specific directions. This quadrangulated the mesh and reduced to the polygon count, but unfortunately, it also destroyed the work I had done to fix the eyeballs. The ends of Sluggo's eyestalks became thin and pointy.

Step 4: Clean up in Mudbox

Sluggo in MudBox

Autodesk Mudbox is a sculpting tool that allowed me to "puff up" Sluggo's eyeballs and give them back the structure lost in the previous step of the process. I then corrected any errors in the texture material by using the stamp tool directly on the model, which works a lot like the stamp tool in Photoshop. I also flattened out the half circles that comprise Sluggo's mouth and refined other parts of the model.

Step 5: Back to Maya

Sluggo in Maya down to 77K+ triangles

The mesh now has a quad topology and is reduced to 77,000+ triangles. That is still a higher polygon count than I think I'll need, but I know that I want extra detail in the mesh so that it can nicely deform as it animates.

Next Step: Riggging

Sluggo the Virtual Puppet: Part 1

Winter 2025 Review: Sluggo the Virtual Puppet

Actual Sluggo meets Virtual Sluggo

I recently completed an interactive project to learn more about MediaPipe, a Google-developed set of open source tools used for computer vision tasks such as face tracking and hand tracking. For this project, I designed and built a physical puppet, digitized it, rigged it, and developed a tool in Unity to control it with hand gestures.

The Physical Sluggo

The first step in this multimedia project was to design and produce a physical hand puppet, affectionately named "Sluggo."

Step 1: Research and Inspiration

Inspirational reading from my home library

I immediately took inspiration from the Henson Muppets for this project, which was a great excuse to peruse some of my books on the subject. Jim and Jane Henson and their earliest collaborators pioneered their own new, distinct style of puppetry, breathing fresh life into an ancient art form.

I settled on a "mitten head" style hand puppet, with the lower jaw operated by my thumb, much like Kermit the Frog. This would be made of a soft material like felt or cloth and the character would be malleable and organic. I decided against adding rod hands (like Kermit) or live hands (like Cookie Monster) as I felt that tracking a second hand may be beyond the scope of this project.

Step 2: Design

Cutting felt for two halves of the head and neck

I found some puppet patterns online that I used as a base for designing my own pattern. I assembled the paper pattern together to make sure that everything fit correctly, then disassembled it before I remembered to photograph it for my process report (oops). I chose brown felt as I originally had in mind to create a dog character, inspired by Rowlf the Dog, the first Muppet to become a "star." I traced my pattern onto the felt and cut out each piece.

Step 3: Production

Sewing the felt halves together

A folded, cardstock oval serves as the puppet's mouth (see photo, above), giving the soft felt a bit of structure. I sewed felt loops to the top and bottom of this piece to better give my fingers a grip when operating the puppet. I used craft glue and a needle and thread to assemble each piece together.

This puppet is expressive, but the head is collapsing

I tried to design pointy ears to add expressiveness to my dog, since it didn't have any arms to move around. My index and pinky fingers fit into these "ears" and I can easily move them in different directions.

The only remaining step was to design a pair of eyes, which I thought I would place on the head much like Kermit the Frog. It was at this point that I realized that the soft puppet was just too soft.

Step 4: Refinement

The "skull" designed to fit inside the head

I quickly cut some craft foam into a "skull" to provide better structure to the puppet's cranium. It needed a couple of holes so that I could fit my fingers through to operate what I still thought of as "ears." 

The puppet was nearly complete, but I had a serious problem. This doesn't look much like a dog! This strange puppet looks more like a slug to me, so Sluggo he must be.

Step 6: Sluggo Lives!

Ready for his close-up

I stole some lightweight, air-drying clay from my kids' craft kit to create the eyeballs, adhered to the eyestalks (formerly, ears) with craft glue. Two black circles with a Sharpie marker completed the process, and a new star is born!

Next Step: Digitization

"Digitization? That sounds like it's gonna hurt!"