Virtual Marionette

Prometeu – Real-time Illusion

Luís Leite and Marcelo Lafontane received an honorable mention at Avanca Festival 2014.

International Conference on Film – Art, Technology and Communication.

The prize Eng. Fernando Gonçalves Lavrador was ax-aequo to the Finn researcher Jouko Aaltonen from Aalto University and to the Brazilian Carlos Júnior Rosa from São Paulo University.

There were two honorable mentions, one to the portuguese researchers Luís Leite and Marcelo Lafontane and the other to the Danish Michael Morgan from the European Film College.

The internationally jury for this prize was composed by academic

International academic Juri commission : Francisco Garcia (Spain), Wai Luk Lo (Hong Kong), Yen-Jung Chang (Taiwan) and by the portuguese Anabela Oliveira, José Ribeiro, Rosa Oliveira, Pedro Bessa and José Marta.

In 24th of July of 2014 I will be presenting the paper Prometeu – Ilusão em tempo real at the Avanca Festival in Portugal.

Leite, L., Redondo, M. (2014) Prometeu, Ilusão em tempo-real. Conferência Internacional de Cinema, Arte e Tecnologia. Avanca.

Prometeu is a multimedia theater play inspired by the Indonesian shadow puppetry, Wayang Kulit. This multidisciplinary project narrows the boundaries between theater and film, past and present, reality and illusion. Merging different art forms brings new experiences to the artists and to the public; in particular the amalgamation of puppetry with film and other disciplines contributes to a new concept, what we can call the performance cinema or ” Live Cinema”.

Using multimedia technologies and cinematic language this project motivates a different paradigm from the traditional reading of the narrative sequence. It promotes the discussion about the boundaries of theater and film, and the role of actors, puppeteers, animators and directors in this particular art form.

Preserving the heritage of traditional shadow theater, Prometeu overcomes the typical physical constraints found in traditional puppet theatre through film and multimedia techniques. Real-time post-production, performance animation or interaction techniques provide a new mean of creation to artists.  By using sensors the artists can expand the interaction. By using live cameras the performer becomes the director creating the sequence in real-time. Techniques such as “luma-key”, procedural animation or gesture tracking enhance the live experience making the performer into a sort of magician.

In Prometeu one puppeteer, performs with virtual and real objects and controls all the show remotely.

In this paper we present the methodology and the implementation of our technology as well as the aesthetics and conceptual aspects. 

LeapMuppet is a digital hand puppet based on the Leap Motion interface.
It was developed with the new skeletal tracking in Unity.

An inspiration on the traditional glove puppet, in particular the muppets, from the great Jim Henson.
This is a digital puppetry proof of concept. The ideia is to bring the art of traditional puppetry manipulation to performance animation.
Not real´istic animation, but believable animation…
Do not believe that the puppet is for real, just believe that the puppet is alive.

A simple character made in Maya, which is real-time animated with the motion of the hand.
I use a skeleton to drive the nose and the head of the puppet (position, orientation), and blend shapes to control the mouth and the eyebrows.
The palm of the hand drives the head (direction free); the index finger drives the nose (direction free); the thumb controls the mouth (up and down); the ring finger controls the left eyebrows (shrink / expand); the pinky finger controls the right eyebrows.

With just one hand you can give great expression to the puppet. Imagine what you can do with two hands ?

Let´s have some fun with virtual puppets.

Live cartoon show for television developed by TV-animation in 1999.

An interactive television show that allows children to call into the show, appear in real-time as an animated cartoon character.

The Cartoon Broadcast System was a custom hardware/software  real-time animation system developed by TV Animation, which allows producers to have real-time lip sync of up to 16 simultaneous sources, live pan, zoom and tracking and electronic puppeteering.

Behind the scenes from Nelly Nuts Show in Portugal which was called “Rita Catita e o Ursinho Oops”. It was produced by Miragem a portuguese production company in a 30 minute daily show  and broadcasted by TVI a portuguese television channel.

Hugo was an  interactive television show created by a Danish company called Interactive Television Entertainment (ITE) in 1990. An interactive game show broadcasted all over the world (more then 40 countries) until 2007. Players at home could play with the character by calling the television show and using their telephone keys as controller. Hugo was developed on custom-made ITE hardware based on a Commodore Amiga 3000 with MIDI devices and DTFM digital converter  for control. Later it was ported to a PC-based system using motion capture for live animation. Body movement and face expression of the puppeteer was retargeted to Hugo virtual character. An early digital puppetry system with great success during 17 years in the television live entertainment.

Motion capture system used to control Hugo character

The ITE 4000 a PC-based system

A MIDI Control panel with a joystick and buttons to manipulate the ONLINE contents.

FlickerLab’s Cartoon Broadcast System (2012)

CBS is an update of the TV-animation Nelly Nut Show animation system.
It brings new features and provides more flexibility for live animation using new controls.
Lip-sync is real-time.

Here is an example of a show called “Café Central” made by a portuguese production house (HOP)

Johnny Bravo Live

In the year 2000, Cartoon networks presented and broadcasted a live cartoon show, Johnny Bravo was a 2D cartoon digital puppet animated in real-time. Johnny Bravo was a kind of “CJ” or cartoon jockey listening to request from telephone for nostalgic cartoons.

Kydara FilmBox On-Air was the first commercial solution to provide a generic real-time animation system using motion capture. It brought more flexibility to the broadcast industry because production houses or television channels could produce their own lives shows.
Producers or animators/puppeteers could trigger 3D/2D animation clip sequences in real-time.
This solution was designed to provide performance and versatility for live broadcasting.

Although the character is not fully live animated it can respond to real-time impulses creating the feeling of interaction with the audience. In a matter of fact, the only thing that is live performance is the lip-sync, it recognizes the sounds spoken by an onset actor and changes the image of the mouth to the most correspondent one.
The puppeteer just press buttons to trigger animations. The animations are previous built from dozens of animated sequences made with hundreds of frames.

The Cartoon Networks used a beta version of On-Air.

Reference: http://www.cgw.com/Publications/CGW/2000/Volume-23-Issue-8-August-2000-/Cartoon-Jockey.aspx

Medialab Paris was one of the very first production houses to explore digital puppetry.

Since 1989 Medialab team as built several dozens of real-time puppets and explore many different interaction techniques. MediaLab open a second house in LA, but what happen to all of this stuff ?

Here is a link to a very interesting documentary about real-time animation and Medialab´s production pipeline, a documentary directed by Pascal Tosi.

The Phenakistocope is a moving image toy or an early animation device developed by Joseph Plateau in 1841.

An early animation device consisting of a disc or drum which rotated, showing successive images through slits, often via a mirror, thus producing the illusion of motion.

‘phenakistoscope’ means  cheat device

phenakisto = cheat

scope = device