Find out how AI is set to transform our approach to special effects by making motion capture accessible to everyone.
Are wetsuits covered in furry balls about to come to an end? As sad as that would be for today’s fashion, the rapid advance of AI might soon make traditional motion capture techniques obsolete. In this blog, we take a look at some of the benefits that an AI-infused VFX world might offer, as well as some of the hurdles that still need to be cleared.
The motion capture landscape
When motion capture (or ‘mocap,’ if you’re a sitting-at-the-back-of-the-bus kind of cool kid) first came on the scene, providing an immaculate way of animating CG characters with almost inch-perfect human movements, it completely transformed the entertainment industry. Today, it’s used in everything from video games and Hollywood blockbusters to the latest wave of VR experiences.
The process is simple. An actor pops on a customised suit that is covered in reflective markers or sensors. As the actor moves around in the suit, multiple cameras track the changing positions of these markers within a defined area, known as the ‘capture volume.’ This real-life positional data is then transferred to the CG animation team so that they can map it onto the CG character’s movements.
Yes, there are still plenty of animation styles which don’t rely on a human avatar, but just think where Marvel Studios would be without the ability to drop the energy of an actor’s performance into their suite of fantastical CG super heroes (we’ll leave you and the rest of the cinema-going public to decide whether that’s been a good thing…).
However, motion capture isn’t perfect:
Cost: High-end motion capture systems can be prohibitively expensive, requiring specialised equipment and vast physical spaces for recording.
Time: Getting the actor kitted up in the suit and properly calibrated takes time, which on set translates to money.
Editing: Cleaning up and processing all of the captured data is seriously resource intensive, and that’s without mentioning the odd occlusion issue where you have to account for a marker that has been obscured from the camera’s view.
How AI can transform motion capture
What makes AI-based motion capture different? Well, unlike the traditional motion capture process, AI mocap can use machine learning algorithms trained to recognize and predict human motion based on video footage. This has a number of advantages:
Dollar bills: With no need to squeeze an actor into a suit and calibrate dozens of different cameras, the whole process is quicker and therefore cheaper.
Simplicity: Although the AI would still prefer you to shoot your actor against a clean background, as the system becomes more sophisticated it will be able to handle more complex environments. This would allow directors to film in situ and drop a CG character animation on top, rather than having to film everything individually.
Clean Plating: clean plating is the process by which an actor is digitally removed from a scene to leave an unobstructed background. It’s normally a lengthy process, but with AI you’ve suddenly got a set of algorithms that can compare the ‘missing’ background elements (blocked from view by the actor) with the rest of the picture, and fill in those gaps. The technology hasn’t been perfected yet, and most AI-generated clean plates still need a bit of human tidying, but the potential is there.
Character Compositing: If clean plating is about removing a character from a scene, compositing does the opposite: it’s the process by which you insert a CG character into real world footage. It’s an absolute art, because whoever is doing the compositing has to meticulously match lighting, color tones, depth of field and a host of other minute visual details in order to ensure both the character and the footage blend seamlessly. But with machine learning, you can give this task to an AI and it will slowly learn to ‘match’ these two different worlds. Whilst you currently still need a human at the end to add the finishing touches, AI can significantly reduce their workload whilst increasing the accuracy and speed of the compositing process.
The future
As things stand, AI models aren’t yet able to replicate the high level of precision achieved through traditional motion capture techniques. But as you can see from the examples above, there are already several areas where AI can reduce the workload for VFX teams.
And what is certain, is that as AI models are increasingly exposed to motion capture workflows, we will see a rapid leap in their capabilities.This, in turn, will have a democratising effect on the industry. As AI is able to shoulder a larger share of the workload, the cost of motion capture will reduce proportionally, opening up new possibilities for smaller studios and productions.
But it will also have a profound impact on our ability to exercise and explore our creativity. Freed from the constraints of traditional motion capture and the rigidity of the ‘capture volume’ space, film makers will be able to employ new parameters within scenes as well as more dynamic camera movements. Actors too will be able to spend less time trying to master the technical aspects of a motion capture suit, and more time focused on their craft.
The result will be a production environment that is more innovative, more artistic and more original.
And at the rate that AI is progressing, we’re pretty sure that it won’t be more than a few years before some young genius who’s only ever worked with AI motion capture turns to us and says “Wait, before this, you used to have to wear what…?
And last but not least, the final result…
