Virtual Marionette

[Links for connecting devices to Game Engines]

PS Move for Unreal 4 (Mac/Win) – https://github.com/cboulay/psmove-ue4

Leap Motion for Unreal 4 (Mac / Win)

Microsoft Kinect 1 for Unreal

Microsoft Kinect 2 for Unreal

PS Move for Unity (Mac/Win) – https://github.com/CopenhagenGameCollective/UniMove

UniWii: Nintendo WIImote for Unity (Mac/Win) – http://wiki.unity3d.com/index.php?title=UniWii

WiiYourSelf: WIImote for Unity (Windows) – https://github.com/lonewolfwilliams/WiiYourselfUnity

Microsoft Kinect (OpenNI)

A Multimodal digital Ecosystem developed for live performances.

A solution for real-time virtual object manipulation, for show control, and for video routing.

for more information about the performance and the POEX project: http://po-ex.net/taxonomia/materialidades/performativas/retroescavadora-sem-titulo

[vimeo 126071525 w=580&h=326]

When you develop contents for live performances relying one just one application you might come across some difficulties in creating specific effects witch are easy to create with other tools.
You might find yourself asking, why can I use multiple applications in a live performance ?
In general the problem relies on how to handle all the video outputs from the different applications without having to switch outputs during the performance and how to control them in a seamless away.

This framework integrates multiple applications sharing one output providing a simple solution to interact and handle data, video and sound. A flexible remote control system that provides scalability and provides the right path for creators to go beyond application limits. It can even handles application crashes because the output is independent from the applications.

Eco-System features
– Shared output
– Shared control
– Flexible remote control
– Scalable data-flow to connect video, audio, and data

This eco-system was developed for the “Untitled” POEX live performance
For this performance we choose to work only with free applications.
Pure Data (puredata.info) a visual programming environment, using chdh Egrégore (chdh.net/egregore_source_manual.htm) an audiovisual instruments interaction environment developed for the egregore performance. The series of patches provides us with a particle system for the creation of an audio visual organic life that would represent the placenta.
Unity (unity3d.com) gave us the right environment for simulating a physical space where the hands of the performer could interact with virtual objects. But Unity does not handle very well typography animation.
Emotion (adrienm.net/emotion/) A real-time animation environment that presents many interesting features for text animation.

Working on a macbook retina 2.4 i5.
NOTE: there is some performance issues during the video capture because further then all the eco-system running I had to open a camera and capture the screen. The sound from this video was not made during the simulation.

Session #04 | 4 April 2015 @ 17h00 > Text-Space [Installation, Performance, Spacial poetry].

Presentation by Rui Torres. Interventions by Nuno M Cardoso with Luís Grifu and Luís Aly.

Live Arquive is Anarchyve! Presentation and remix of the Digital Arquive of Portuguese Experimental Literature (www.po-ex.net)  by Retroescavadora – Artistic Intervention Collective: Ana Carvalho, Filipe Valpereiro, Luís Aly, Luís Grifu, Nuno Ferreira, Nuno M Cardoso, Rui Torres.

At Gato Vadio bookstore

The Digital Hand Puppetry prototype was presented at Mostra UP. Participants were able to experiment the manipulation of virtual puppets with their hands using a Leapmotion controller.

The digital hand puppetry prototype attracted many participants.

Interesting insights from the participants made me think in other directions for this project.

Most of the participants were able to manipulate the puppets in a few seconds without any help. Some of the participants shacked their hands and flexed their fingers with rapid movements, others went for a slow approach. When using their left hand, participants become surprise with the mirror movement of their hands and with the physic simulation interaction that opens a different approach of interacting with the puppet and the props. Like a real hand making adjustments.

On the 2th of March we will give a lecture on the puppet play “Prometeu” at the school Clara de Resende in Porto.

Picture by Pedro Martins

Almost all the production team will be present at the lecture:

Puppeteer: Marcelo Lafontana

dramaturgy: José Coutinhas

Photography: Pedro Martins

Multimedia: Luís Grifu

Scenography: Sílvia Fagundes

Luís Leite presented in the 24th of February the Inversus publication at the Eighth International Conference on Advances in Computer-Human Interactions (ACHI) in Lisbon.

This publication is now available for download at the download section.

Digi Mario – A digital rod puppet style with physics. A hand manipulated puppet with leap motion controller. One hand controls the puppet and the other simulates the puppeteer virtual hand for physical interaction.

[vimeo 110798138 w=580&h=326]

This video shows how to animate a puppet with just one hand using physics to recreate the marionette aesthetics.

Marionette animation is fascinating but requires a lot of skills, this virtual marionette is much simpler but the digital puppeteer can control many aspects of the puppet. With few training the puppeteer can bring this puppet to life in a traditional style.
All fingers are mapped to the face controls and the hand is mapped to the head. When you move the head, the body will follow along like if we had a rod connected to the head.
Interesting effect is when you turn your hand using your pinky finger first, the eyes will look to the target and the head will follow the eyes.

framework: Leapmotion v2+ Unity

[vimeo 110452298 w=580&h=326]

Demonstration of full hand control for expressive digital puppetry

This video shows how to animate a digital puppet in real-time with just one hand in an expressive manner.
It is part of a PhD research in the digital puppetry field.

Stringless hand controller (a metaphor for the marionette controller)
Hand + 5 finger controls different aspects of the puppet

Hand: position and orientation of the puppet
Pinky finger: Eye Pupils rotation in all directions
Index finger: Eyelashes rotation in the +Y and -Y axis (open and close)
Middle finger: Right Eyebrow blend shape deformation for character expressions
Ring finger: Left Eyebrow blend shape deformation for character expressions
Thumb: Mouth (open and close) blend shape deformation

There are different degrees of freedom (DOF) for each finger mapped to a certain puppet control.
The Hand as 6 DOF (position and rotation), you can move your hand freely around the tracking area but must be carful with the occlusion problem;
The pinky finger has 3 DOF for rotation of the eye pupils, and although controlling the pinky finger independently as some constraints, for this kind of small motion is more then adequate.
The middle and ring fingers are have more constraints and it is very hard to controlled them independently, so I mapped just 1 DOF (up and down) of each finger to the eyebrows.
The index finger as more potential because you can control more degrees of freedom, I tried using the index to control the eye pupils but the results were not so great. Instead, the index finger is mapped to the eyelashes rotation with just 1 DOF (up and down)

This is a good direction for digital puppeteers that have full control of the character expressivity. It require some training to act like a character, but that´s the magic of puppeteering.

This prototype can animate two different full controlled puppets for interaction.
A powerful model for performance animation using digital puppeteering techniques.

Hardware: Leap motion device for hand tracking and a macbook
Software: Unity Engine

The Porto International Marionette Festival (FIMP) is out there again showing the diversity of puppetry from all the world.

Although the puppetry plays are quite interesting I´m delighted with the documentary film cycle called the Marionnettes du Monde from Acrobates Films where I had the opportunity to learn more about puppetry in Taiwan and Poland.

Marcelo Lafantona won the award for best male performance with Prometeu at the 16th International Puppetry Festival in Varna – Bulgaria.

The renewed Prometeu system was performed with great success by the hands of Marcelo. A great performance presented to the Bulgarian audience. The Juri although not knowing the spoken language as chosen Marcelo´s performance as the best male performance.

The new system is more stable and flexible helping Marcelo to be more focus in his performance. Marcelo is the one man show, controlling lights, sounds, video and animation and all the puppeteering. And although it was very complex to create all the stuff is changed by the puppeteer in real-time with just one button.

A very important recognition from one of the oldest Eastern Europe festivals with a great tradition in the art of puppetry.

Congratulations to Marcelo Lafontana.

Photos from TMPC – Varna, Stage Filial – Drama Theater Stoyan Bachvarov