Digital Puppetry System Development

[Andrew]

Hi everyone, guess what I've been working on?

About a year and a half ago I started looking in to Machinima and digital puppetry and thinking about ways it could be improved/developed. After speaking to some people who have done this kind of work and doing a fair bit of research I started developing an open source digital puppetry system. Essentially, when it's completed it will allow puppeteers to control 3D digitally animated characters in real-time. The whole system is being built around Blender the free open source 3D software so there's no need for expensive high end programs that cost thousands of dollars.

After about two months of full time work I'm close to have a usable control system and tonight I finished putting together the basic head controls. There's a quick demo video available at http://youtube.com/watch?v=z-SS8JFtUL4

It's not much to look at for now, but if you use your imagination I think this kind of thing has a lot of potential once it's further developed.

The head is controlled using a joystick, but you can use any kind of input device (mouse, keyboard, dataglove, even a Wiimote!) to control the character on screen. The head is just a basic monkey head without any details or shaders (textures) for now.

I've got some background info on this project in my Machin-X blog for anyone interested.

[Magenta]

How very awesome!  I love this concept.  What a thing to think of.  Talk about going in a whole new direction. 

Does the realtime part of it need a really good processor?  I imagine the more detailed your graphics get the harder it works to keep up.  Of course, a professional would get set up for that. 

I do love this!

Meg

[Sonny]

Incredible Idea. This should really be presented to video game companies. If this the first of its kind, I see video game compatibility. Very nice concept and I hope we get the demo version before the public.

Thanks Andrew

Your computer rendering looks like my EX

[Billy D. Fuller]

Andrew

You've done it again, very nice work. What a different angle to add to puppetry. Thank you so much for always stopping by and sharing your new adventures with us. It will be interesting to follow your development of this project. Please keep us updated.
Well it has been interesting following all your projects, did you ever get around to designing your puppet pattern?

Thanks
Billy D.

[Andrew]

Thanks guys!

This runs fine on my rather modest laptop (1.4 GHZ, 480mb RAM, no 3D card), although the demo is using only 500 polygons (more polygons = more detail, but also requires more processing power) and no textures.

In theory you can do graphics that are near Pixar-quality this way, but that requires a workstation that costs between $10,000-20,000 and Blender's real-time graphics have some serious limitations right now, but its developers are in process of addressing that.

This Bad Language Mario video has a good example of a digital puppet "talking head". The Disney theme parks have an attraction called Turtle Talk With Crush where Crush the Sea Turtle from Finding Nemo appears in an "aquarium" - actually a big video screen - and interacts with the audience. Both of these use the same basic principles I am working with.

What's really cool (I think anyway) about the approach I am trying is that because it's tied in to Blender you can use all of Blender's animation tools too, which are really powerful. You can perform in real-time, then edit your performance and correct any mistakes with Blender before rendering the final work out like conventional 3D animation that's very high quality.

I'm about halfway through setting up a bunch more features on the head, I'll make up another video and post it as soon as I am done.

[Sonny]

Quick, Some get him coffee!!! 

[Shawn Sorrell]

Ahhh... man now I am going to have to get past that learning curve thing and open up Blender.  For some reason I just can't seem to be able to get the hang of 3D yet.  

Great job Andrew!  You seem to always accomplish what I "want" to do. 

[StiqPuppet Productions]

Great job!  I love your Blog also I check it every day!!

Thanks for challenging our brains with your new ideas and concepts!!

Keep it coming and update us on this!!

Daryl H.

[DrPuppet]

Nice Andrew so essentially you made a low teck waldo unit like the Muppets used on the old Jim Henson hour to cntrol their digital puppet guy Steve Whitmire did. Nice work though! I can't imagine designing something like that

[Andrew]

It's pretty much the same idea, except it's software not hardware. The system is what I like to call "controller agnostic" meaning that even right now anything you can plug in to a computer (even a waldo) can be used to control an on screen character. I want the system to adapt to puppeteers, not force puppeteers to have work with a specific type of controller.

Once the control system is done I want to add colour and motion tracking too, so eventually you'll be able to perform a physical puppet on a webcam and the onscreen character will move accordingly. Students at the University of GA already have a system like this up and running for Machinima called (appropriately enough) Puppet Show.

[Ron G.]

Sweet!



Ron G.

[Andrew]

Here's another video, this one demonstrates some extremely basic lip sync capability -Video Removed By YouTube

Phonemes (a-i-e-o-u mouth shapes) aren't possible yet because I have to figure out some workarounds for the somewhat limited bone animation system in Blender's GE. I actually had the face almost completely set-up for full animation yesterday, but it couldn't turn or tilt.

Work on this will continue through the weekend and I'll post again when I have more to show.

[Billy D. Fuller]

Thanks again Andrew, you are determined to breathe life in to this beast aren't you. 
Keep at it, , just be careful and don't let things get out of hand. You may be responsible for the next King Kong and he will carry you away......... 

Just a little humor there, you're doing a fine job and it is coming along great. Thanks again for sharing your progress with us. It interesting to follow your work. You are yet another inspiration to the puppet community.

Billy D.

[Andrew]

Thanks Billy. I pulled down the video at the link above because I forgot to add two additional clips.

Here's the new link - http://www.youtube.com/watch?v=ZxkFpoHUfnM

The second part of the new video shows an articulated jaw with a very crude squash/stretch effect. The third part demonstrates a keypose that's triggered by pressing a joystick button. In the video the keypose is just a movement of the monkey's snout (it was done quickly and is really bad actually), but the keypose could be any movement or animation cycle that you wanted. For example it could be programmed so that a scream or a laugh happens when you press a button.

I also smoothed out the surface of the model so it looks a little nicer now.

[StiqPuppet Productions]

Really impressive Andrew!

It really has come along since the first time you showed us your test!! I like the "skin" tests, you can really start seeing a character come through.

Can't wait to see more!!

Daryl H.