Virtual Marionette

I had the opportunity to attend to the 10th International Shadow Theater Festival in Schwäbisch Gmünd, Stuttgart, Germany.

The festival took place in the city of Schwäbisch Gmünd that every three years turns into the most important meeting for the contemporary shadow theater. Shadow theater companies from all over the world were represented: Portugal, Brasil, England, USA, Argentina, France, Italy, Poland and many others.

From the early cave drawings that shadows have been a part of the human communication. The human being is fascinated by the shadow expression as something that as its own life and explored by the mythology. The shadow player is like a shaman that guides our fear from the shadow of the death. The shadow play is connected to spirits, gods and demons or mythical creatures. But the light source as suffer considerable upgrades in its technology. The light from the fire as been replaced by lamp bulbs and projectors generating sharp shadows, and now, with the introduction of halogen lamps it is possible to use it as a performing light, a point light in the stage operated by the performer to create spatial effects. It allows to change the perspective, and create motion and image blending as in film. The last upgrade to this artistic form can be consider somehow problematic and is creating some resistance to recognize it as a shadow technique, the use of cameras and computers.

This year program presents diverse approaches to the shadow theater – a bridge between the traditional form and the contemporary shadow theater.

Last week (8th Oct.) attended the marionette workshop at Museu da Marioneta in Lisbon.

This workshop was oriented by Marcelo Lafontana and Bernd Ogrodnik.

Bernd Ogrodnik his a great puppet master, and was the designer and lead puppeteer for the Marionette Movie “Strings”, currently he is the Artistic Director of the Icelandic Center of Puppetry Arts.

It was a great honor to listen to the Bernd Ogrodnik and to learn how to respect puppets, and to understand how important it is the concept of awareness in puppetry.

Common spaces poster was available during the 2 days of the 1st Joint Conference and Exhibition of the International Partnerships. I have the opportunity to demonstrate the framework of Common Spaces during the conference with a simple setup.

This conference highlights the Portugal-U.S. partnerships. A joint conference and exhibition on fostering science and innovation, a showcase activities and outcomes of the international partnerships with Portugal.

First experiments with DIY holographic projection with digital puppetry.

Ravin Balakrishnan already in 2001 was exploring physical mockups to interact with 3D volumetric displays (User Interfaces for Volumetric Displays –
http://www.autodeskresearch.com/publications/volumetricui). Naiqi Weng made some interesting experiments with volumetric displays and digital puppetry in 2005. Andrew Jones extended this approach in 2007, (Rendering for an Interactive 360º Light Field Display- http://gl.ict.usc.edu/Research/3DDisplay/)

But this is a completely different approach, a low-cost solution that can be easily done with cheap materials and a standard computer display.

It is a four-sided hologram made with transparent plastic that sits above a computer screen. The screen is divided into four planes that reflects the performance animation manipulated by the digital puppeteer with his hands.

This experiment was made in PIC – Porto Interactive Center and the holographic structure was developed by Marcelo Gomes.

To learn more about four-sided pyramid holograms jump to: http://rimstar.org/science_electronics_projects/hologram_pyramid_diy_homemade.htm

[Links for connecting devices to Game Engines]

PS Move for Unreal 4 (Mac/Win) – https://github.com/cboulay/psmove-ue4

Leap Motion for Unreal 4 (Mac / Win)

Microsoft Kinect 1 for Unreal

Microsoft Kinect 2 for Unreal

PS Move for Unity (Mac/Win) – https://github.com/CopenhagenGameCollective/UniMove

UniWii: Nintendo WIImote for Unity (Mac/Win) – http://wiki.unity3d.com/index.php?title=UniWii

WiiYourSelf: WIImote for Unity (Windows) – https://github.com/lonewolfwilliams/WiiYourselfUnity

Microsoft Kinect (OpenNI)

A Multimodal digital Ecosystem developed for live performances.

A solution for real-time virtual object manipulation, for show control, and for video routing.

for more information about the performance and the POEX project: http://po-ex.net/taxonomia/materialidades/performativas/retroescavadora-sem-titulo

[vimeo 126071525 w=580&h=326]

When you develop contents for live performances relying one just one application you might come across some difficulties in creating specific effects witch are easy to create with other tools.
You might find yourself asking, why can I use multiple applications in a live performance ?
In general the problem relies on how to handle all the video outputs from the different applications without having to switch outputs during the performance and how to control them in a seamless away.

This framework integrates multiple applications sharing one output providing a simple solution to interact and handle data, video and sound. A flexible remote control system that provides scalability and provides the right path for creators to go beyond application limits. It can even handles application crashes because the output is independent from the applications.

Eco-System features
– Shared output
– Shared control
– Flexible remote control
– Scalable data-flow to connect video, audio, and data

This eco-system was developed for the “Untitled” POEX live performance
For this performance we choose to work only with free applications.
Pure Data (puredata.info) a visual programming environment, using chdh Egrégore (chdh.net/egregore_source_manual.htm) an audiovisual instruments interaction environment developed for the egregore performance. The series of patches provides us with a particle system for the creation of an audio visual organic life that would represent the placenta.
Unity (unity3d.com) gave us the right environment for simulating a physical space where the hands of the performer could interact with virtual objects. But Unity does not handle very well typography animation.
Emotion (adrienm.net/emotion/) A real-time animation environment that presents many interesting features for text animation.

Working on a macbook retina 2.4 i5.
NOTE: there is some performance issues during the video capture because further then all the eco-system running I had to open a camera and capture the screen. The sound from this video was not made during the simulation.

Session #04 | 4 April 2015 @ 17h00 > Text-Space [Installation, Performance, Spacial poetry].

Presentation by Rui Torres. Interventions by Nuno M Cardoso with Luís Grifu and Luís Aly.

Live Arquive is Anarchyve! Presentation and remix of the Digital Arquive of Portuguese Experimental Literature (www.po-ex.net)  by Retroescavadora – Artistic Intervention Collective: Ana Carvalho, Filipe Valpereiro, Luís Aly, Luís Grifu, Nuno Ferreira, Nuno M Cardoso, Rui Torres.

At Gato Vadio bookstore

The Digital Hand Puppetry prototype was presented at Mostra UP. Participants were able to experiment the manipulation of virtual puppets with their hands using a Leapmotion controller.

The digital hand puppetry prototype attracted many participants.

Interesting insights from the participants made me think in other directions for this project.

Most of the participants were able to manipulate the puppets in a few seconds without any help. Some of the participants shacked their hands and flexed their fingers with rapid movements, others went for a slow approach. When using their left hand, participants become surprise with the mirror movement of their hands and with the physic simulation interaction that opens a different approach of interacting with the puppet and the props. Like a real hand making adjustments.

On the 2th of March we will give a lecture on the puppet play “Prometeu” at the school Clara de Resende in Porto.

Picture by Pedro Martins

Almost all the production team will be present at the lecture:

Puppeteer: Marcelo Lafontana

dramaturgy: José Coutinhas

Photography: Pedro Martins

Multimedia: Luís Grifu

Scenography: Sílvia Fagundes

Luís Leite presented in the 24th of February the Inversus publication at the Eighth International Conference on Advances in Computer-Human Interactions (ACHI) in Lisbon.

This publication is now available for download at the download section.