A Project by Danyi LIU ,Felipe Astolfi & Nana Tian
Chinese Shadow Puppetry or Pi Ying Xi is an ancient method for storytelling. Originated from 2000 years ago, people used leather to make different shapes of the characters, with an oil lamp, the shadows moved on sticks. Because of the special techniques such as improvisational singing, falsetto, simultaneous manipulation of several puppets, and the ability to play various musical instruments, shadow puppetry was inscribed in 2011 on the list of the Intangible Cultural Heritage of Humanity.
Before the coming of television and movies, shadow puppetry was a popular form of entertainment for both lord and loon, young and old. Accompanied by music and singing, puppeteers brought laughs and tears to the audiences with imagination. However, this traditional folk art is losing its audiences. Even when you go to a museum which exhibits puppets, there might be a big screen playing documentary videos, not attractive at all.
When it comes to shadow puppetry, we will never miss these four parts - character design, control mechanisms, live sound manipulation, and storyteller. Our group project Silhouette Theater is a system which can provide people a new way to experience the live performance of shadow puppetry in order to raise their interest.
We use SimpleOpenNi, a Processing library to get the raw 3D joints data (x, y, z) of the Skeleton, then convert the data into 2D (x, y). The filtered data is divided into two parts - a) sent into Max/MSP via OSC(port 7220) to control the live music, b) sent into Animata via OSC(port 7110) to control the puppet (see below), but we have tested many times before that in order to get a range of offset and scale to match the coordinate in Animata.
2) 2D Graphic Design
Since we are using the skeleton data from Kinect to control the characters, we need to design a corresponding skeleton with all the main joints (head, neck, torso, shoulders, elbows, hands, knees, feet) of a character at first. Then we draw different parts of body separately (head, arms, shoulders, hands, hip, legs, feet) in AI. We can attach all these parts onto the bones in Animata. There are two stylistic choices for the method of applying the 2D texture to the skeleton, with a simple or complex mesh. The simple mesh, the puppet behaves exactly like a paper cut puppet where its parts are drawn apart and mounted in the software. The complex mesh (as the image above) uses a single illustration and create a detailed texture mesh that behaves more like a cloth. We also design both background and foreground to provide a better visual experience to the participants.
3) Sound Design
In order to create immersive stories, we chose Chinese traditional folk music as the background music. We reproduce the music to fit the scene better, and the distance between Kinect and the participant will influence the volume of the background music - the closer the participant is to the Kinect, the lower the music is - the process of fade in can be recognized while a participant enters the scene (fade out - leave). On the other hand, we edit sound samples recorded from different Chinese traditional percussions, the movement of the character can trigger different sounds which may have various effects on the development of a story.