Virtual Marionette

[vimeo 110452298 w=580&h=326]

Demonstration of full hand control for expressive digital puppetry

This video shows how to animate a digital puppet in real-time with just one hand in an expressive manner.
It is part of a PhD research in the digital puppetry field.

Stringless hand controller (a metaphor for the marionette controller)
Hand + 5 finger controls different aspects of the puppet

Hand: position and orientation of the puppet
Pinky finger: Eye Pupils rotation in all directions
Index finger: Eyelashes rotation in the +Y and -Y axis (open and close)
Middle finger: Right Eyebrow blend shape deformation for character expressions
Ring finger: Left Eyebrow blend shape deformation for character expressions
Thumb: Mouth (open and close) blend shape deformation

There are different degrees of freedom (DOF) for each finger mapped to a certain puppet control.
The Hand as 6 DOF (position and rotation), you can move your hand freely around the tracking area but must be carful with the occlusion problem;
The pinky finger has 3 DOF for rotation of the eye pupils, and although controlling the pinky finger independently as some constraints, for this kind of small motion is more then adequate.
The middle and ring fingers are have more constraints and it is very hard to controlled them independently, so I mapped just 1 DOF (up and down) of each finger to the eyebrows.
The index finger as more potential because you can control more degrees of freedom, I tried using the index to control the eye pupils but the results were not so great. Instead, the index finger is mapped to the eyelashes rotation with just 1 DOF (up and down)

This is a good direction for digital puppeteers that have full control of the character expressivity. It require some training to act like a character, but that´s the magic of puppeteering.

This prototype can animate two different full controlled puppets for interaction.
A powerful model for performance animation using digital puppeteering techniques.

Hardware: Leap motion device for hand tracking and a macbook
Software: Unity Engine

LeapMuppet is a digital hand puppet based on the Leap Motion interface.
It was developed with the new skeletal tracking in Unity.

An inspiration on the traditional glove puppet, in particular the muppets, from the great Jim Henson.
This is a digital puppetry proof of concept. The ideia is to bring the art of traditional puppetry manipulation to performance animation.
Not real´istic animation, but believable animation…
Do not believe that the puppet is for real, just believe that the puppet is alive.

A simple character made in Maya, which is real-time animated with the motion of the hand.
I use a skeleton to drive the nose and the head of the puppet (position, orientation), and blend shapes to control the mouth and the eyebrows.
The palm of the hand drives the head (direction free); the index finger drives the nose (direction free); the thumb controls the mouth (up and down); the ring finger controls the left eyebrows (shrink / expand); the pinky finger controls the right eyebrows.

With just one hand you can give great expression to the puppet. Imagine what you can do with two hands ?

Let´s have some fun with virtual puppets.

TheToyWorlds was evaluated at a junior school as part of the virtual marionette research.

Picture 1 – Childrens interacting with the game at the classroom.

Introduction

We challenged children under the ages of 5 and 6 to grab a virtual puppet and play with it.

The main goal was to better understand how children under these ages who never had contact with digital interfaces based on body motion respond to the interaction. The ToyWorlds is a two player game where the player controls the virtual marionette with their hands. The left hand manipulates the body of the puppet and the head follows the right hand motion. The players grab the puppet with the left hand and move the target where the puppet is looking with the right hand. In this way we challenge the participants to control their own body as if it was a marionette controller. This experience is somehow near to how puppeteers manipulate with their puppets, using rods, strings, or other kind of controller.

Picture 2 – Simulation of the ToyWorlds puppet manipulation using the two hands

In puppetry the head of the puppet is essencial to transmit emotions and to help the audience to understand the direction of the action or motion. By fowling the players right hand with their heads, the virtual puppets call the attention of the audience and a contribution to a more expressive animation. Although this is a very simple way to control virtual puppets by just using our two hands it is a very challenging interaction because our hands are not visually represented in the scene. In this way, we must calculate the position of our hands by the position of the puppet´s body and head target. And because the result does not mirror our body exact motion, we need the capacity to abstract from the real motion to the represented motion. This indirect manipulation creates a distance between the player and the puppet witch we want to understand.  This is one important  aspect we want to explore, study and measure.

picture 3 – The right hand makes the puppet to rotate. The puppets head follows the players right hand forcing the body to rotate.

By assigning two different tasks to our two hands we challenge the players to take control of their bodies, an abstraction of the body which may be very difficult to childrens under 6 years old.

The virtual marionettes are rag-doll puppets, dynamically affected by gravity and collisions, the result is a very natural and rich animation.

picture 4 – screenshot of the game

 From inside out, the game narrative!

In Marta´s bedroom, there are strange things happening when she goes to sleep. She dreams with paper boxes that fill her room creating a big mess. Because she is a little lazy to put her bedroom in order, she created two little imaginary friends that help her to clean up. Joaquin wears a green shirt and his the uncle of António which wears a orange shirt, both are paper figures and very hard workers. They can only grab the boxes when they change their colors, Joaquim grabs the green boxes and António grabs the orange boxes. When the boxes change to the blue color, both puppets can pick them by just looking at them using their respective looking target square. Green and orange boxes are grabbed with the bodies of the puppets using the players left hand and the blue boxes are picked up with the looking target using the players right hand. After grabbing the boxes the player receives a score of one point to the green or orange boxes and ten points to the  blue boxes. The boxes disappear if grabbed or return to their natural color after a certain time.

The players give life to this two little figures and have one minute and an half to grab the boxes. They start by making the calibration pose and the first to get their pose calibrated takes control of the first player witch is Joaquim.

This game evaluates body motor control, an important aspect for the puppetry manipulation.

In the table above we show the game results of this experiment. Two of the 22 children were not able to grab any color box because they felt stressed by the competition.

[table id=1 /]

From the 10th till the 23th of  May at Mosteiro São Bento da Vitória, Porto.

Coproduction Lafontana – Animated Forms / Teatro Nacional São João (TNSJ) from the original text of Fernão Mendes Pinto, with the interpretation of Marcelo Lafontana.

Inspired by the adventures of Fernão Mendes Pinto, a portuguese explorer, reported
from the book “Peregrinação” , published in 1614, Marcelo Lafontana makes a journey through strange stories that are presented in a miniature world.

Through the intersection of puppetry, in particular, the expressiveness of paper theater with the cinematic language, this small world becomes a large space of illusion in which the narrative arises.

The stage, is transformed into a film studio, where scenarios and characters drawn and cut from cardboard are handle in front of video cameras. The images are captured by a multimedia architecture providing image processing, image mixing, sound effects, chroma key, performance animation, and all happens in real-time. The final result is projected on a screen, like a sail of a boat that opens to the charms of a journey through the imagination.
from
Fernão Mendes Pinto

staging e interpretation
Marcelo Lafontana

dramaturgy
José Coutinhas

scenography
Sílvia Fagundes

design of the characters and scenarios
Luís Félix, Rebeca das Neves

photography
JPedro Martins

music
Eduardo Patriarca

Multimedia (architecture and contents)
Luís Grifu

Lighting design
Pedro Cardoso

staging assistance
Rita Nova

I´ve implemented some of the digital puppetry methods in this project. Methods and tools are described in another post in this blog.

Street Outdoor

Music & Emotions at CLIP – Porto International School by INAUTISMO

An event to raise awareness of autism and the research that is being made in this field. An opportunity to present activities and projects for therapy or for stimulating people with spectrum disorder.

“Paralelamente aos workshops, contamos com convidados especiais que vieram alegrar ainda mais este evento, os palhaços do O””pital com as suas brincadeiras, a deficiwood com artigos em madeira feitos à mão e a universidade do porto com o projecto “Virtual-Marionette”, um espaço de exploração da animação performativa (tempo real), recorrendo a interfaces digitais recreando os tradicionais controladores de marionetas.” Reference

ToyWorlds is an artistic installation, a body motion game for two players that stimulates the use of our body in particular our hands to manipulate puppets like marionettes. The player must control a virtual marionette and try to catch boxes with the puppets body or by looking at the boxes.

Structure of the puppet´s booth

A preview of the game.

Exploring Leapmotion interface using hand / fingers to control a puppet´s head.

Very simple example of digital puppetry build in Unity.

(more…)

The Virtual Marionette installation “BePuppit” in a puppetry exhibition at Teatro Municipal de Vila do Conde.
We invite the audience to participate in this play which recreates a traditional puppet booth.

“PUPPIT” is an interactive experience that explores different ways to control and animate virtual puppets in live environments using digital puppetry methods as a storytelling tool.
With “Body-PUPPIT” the users body is the puppet controller, producing performance animation with body motion. We challenge the participants to explore their bodies to give life to puppets that are different from human morphology as if they were using hand shadows to create imaginative figures.
By playing with puppets in a dramatic way the user can create the illusion of life and build a narrative.

PUPPIT consists in a several prototypes that are the work-in-progress of a PhD research called “Virtual Marionette – Interaction Model for Digital Puppetry”. The goal of the prototypes is to explore and evaluate different rigging, mapping and interaction methods, and to develop tools for artists and non-expert artists that can be used in as collaborative environment for storytelling – making live narratives.

The exhibition is open every day since the 14th till 22th December except mondays.

www.virtualmarionette.grifu.com
December 2013

Results from the explorative work made at “Espaço do Tempo” in Montemor-o-Novo in December 2013.

The manipulation of sound and visual elements using the body as interface like a marionette controller.
We explore the potential of the body motion in three main areas:
– Body expression to model sound and visual
– Body motion to trigger sounds
– Body position in space to model and trigger sounds

We define a sensitive space where the performer could move and interact with the audio-visual elements.

As we can verify in the video the way the performer interacts is somehow resembled  to the way a puppeteer manipulates marionettes.

As future work we will explore deeply the perceptive motion of sound and visual. Creating a relation between diegetic and non-diegetic sounds and similar approach to the performance animation.

Video performance by: Amândio Anastácio
Multimedia: Grifu

Dec.2013

Performance Animation LAB. at Cinanima 2013 (Espinho/Portugal) – Animation Film Festival

The main objective of this Lab is to disseminate Digital Puppetry as a methodology to create performance animation.
We present our methodology based on simple steps: Create->Rig->Map->Animate

November 2013

Reel with CINANIMA 2013 labs

(more…)

Result animation from the performance animation Lab at CINANIMA 2013 (Animation Film Festival in Espinho/Portugal).
Real-time storytelling and live directing with virtual marionettes using digital interfaces to create performance animation.
This play was performed with three digital puppeteers / actors and one director.

Framework:
Hardware used: 2 Wiimotes and 2 Nunchucks; 1 Microsoft Kinect; 1 Ipad, 1 Macbook DualCore 2005; VideoProjector
Software used: Animata, Pure Data, Osceleton, Osculator, TouchOSC.

November 2013
www.virtualmarionette.grifu.com