May This Be My Life

The aim was to employ Character Creator and Move.ai for 3D modeling in order to craft a lifelike digital twin of our protagonist. This digital persona would not only focus on the audience but also provide them with an interactive object to engage with.

You appear in a dark space. There are contours of a large wall in front of you. A singer appears behind the wall, holding a small, glowing point-cloud ball in her right hand. The particles within the ball emit a bright light, resembling a swarm of fireflies. She is singing. She flings the illuminating ball into the darkness, where it ricochets off an unseen surface. Briefly, a fragment of a structure is exposed before disappearing in a blaze of light and sound. A flash of music starts and then fades out again as the pixels disappear.

A new ball appears in her hand. She stretches her hand through the surface of the mirror and gestures for you to take the ball. The part of the arm that protrudes from the mirror surface is in black and white hues. She signals for you to dispose of your fireball into the abyss as well.

When you do this, the particles bounce against a wall, get absorbed by it, and a wave of particles flows through the walls, building up the pixels of a large, church-like space. The music continues; the particles in the wall move in sync with the frequency of the music. The acoustics of the voice adjust to the new space.

In a mesmerizing reversal, the church-like area dematerializes back into the point-cloud ball, which then transforms back into the woman’s hand.

On May 24th, we conducted the final capture day for this project, during which we recorded all the necessary material to develop a digital twin of our singing protagonist, Viola. We employed Move.ai, facial capture, video capture, and photography for this purpose. In the subsequent weeks, we utilized Reallusion’s ‘Character Creator’ program to process this material and generate a virtual, interactive character.

Initially, the 3D model of the digital twin was crafted based on the captured material of Viola, with the assistance of a 3D animator. Later, we refined the motion capture data, obtained through Move.ai, to seamlessly integrate lifelike movements into the 3D model.

Move.ai: In addition to the Rokoko motion capture suit, we also utilized a new AI-based solution called Move.ai. To set up Move.ai, we positioned phones around the actor and initiated recording with a remote iPad. Prior to the performance, the actor stood in a U-pose and clapped three times. Subsequently, we specified the timecode for the first and last claps to enable automatic processing on Move.ai’s servers.

Cleanup

A substantial amount of data is captured by Move.ai, providing us with a wealth of flexibility and detail. However, this comes at the expense of cleanup time. The cleanup process involves the following steps:

– Optimizing Frames: Initially, we selected all keyframes for all bones, resulting in approximately 40 million keyframes. We then optimized them to reduce the count to around 40,000. For specific bones, such as the hands, we took additional measures to further reduce the number of keyframes.

– Smoothing Frames: To reduce jitter in the animation, we smoothed out all animation curves between keyframes. For hand data, we used higher values to ensure more natural finger movements.”

Flexibility as a Fundamental Attribute

What we learned during this research is how incredibly vital flexibility and creative thinking are when integrating the latest technological advancements into artistic work. As there are no established workflows or production processes for new technologies, one often works with technology in the beta phase. This means the software is still under development, and errors occur frequently, leading to potential time and monetary losses. Such risks can only be afforded within an experimental trajectory. Our initial idea was to work with interactive volumetric video. After an interesting and intensive year, we concluded that the magic lies in the combination of using volumetric video and software to develop a digital twin. Despite taking a detour, we made significant progress with our research question.

The Pace of Technology

Even for us as technology enthusiasts, it’s fascinating to witness the rapid advancement of technology. We find ourselves amidst a ‘virtual revolution’ where AI is assuming an increasingly significant role in the creative process. These developments are precisely in line with our research objectives, prompting us to eagerly integrate and test the latest technologies. To accomplish this, we collaborated with various experts and consistently adjusted our team to tackle new challenges. Operating in such a swiftly evolving domain also necessitates the ability to swiftly adapt to changes, particularly with regard to team members.

Sprints as an Ideal Workflow

Working in ‘sprints’ proved highly effective for us. It enabled us to establish new starting points consistently and assemble a fresh team within well-defined boundaries. Sufficient time between the sprints/showcases was allocated for contemplating the next steps and conducting thorough technical research.

Technology in Service of Vision and Narrative

Understanding and applying the latest technologies in a creative process demands significant mental capacity. One must continuously reinvent the wheel and confront new challenges each time. It’s crucial to regularly assess the extent to which technology contributes to the vision and narrative that need to be conveyed. Ultimately, technology, as we employ it, always serves the artistic vision.

The Importance of Partnership

The hardware and software utilized in this project are quite expensive. This cost factor is attributed to the commercial orientation of software and project developers, leading to the sustained high development costs. Our partnerships, particularly with Scatter and Reallusion, have proven to be highly beneficial. It’s satisfying to observe that our partners have also derived value from our research. Our feedback frequently resurfaced in the form of new software versions. We engaged in collaborative efforts encompassing artistic, technical, and content-related aspects.