Remedy’s not usually one of the first names you associate with top-notch facial animation. These were the culprits behind Max Payne’s perma-strained face, looking for all the world like he’d just prolapsed. Still, Alan Wake and Quantum Break were alright. Still seeking to atone for Max’s unfortunate expression, Remedy has teamed up with Nvidia to work on dramatically improving motion capture and facial animations using deep convolutional neural networks.
Their new system drastically reduces the amount of time and effort needed to create lifelike facial animations for characters, converting real footage of a person into gameplay with 5-10 minutes.
Remedy and Nvidia have achieved this by creating a real-time deep learning framework for facial performance capture, firstly capturing footage using multi-view tracking to provide a 3D image. A convolutional neural network then trains itself to identify self-occluded regions and then produces a high-quality output render. Perhaps most impressive of all, once the neural network has a detailed profile of an actor, it can even render realistic facial animation using an audio clip alone.
What once took hours could now take minutes, which would be a major boon in delivering lifelike facial animations to characters. With the current slow process, it can be too much of a time sink for all but the largest devs, and this advanced technology could speed up the process dramatically.
"Based on the Nvidia Research work we've seen in AI-driven facial animation, we're convinced AI will revolutionise content creation," said Antti Herva, lead character technical artist at Remedy. "Complex facial animation for digital doubles like that in Quantum Break can take several man-years to create. After working with Nvidia to build video- and audio-driven deep neural networks for facial animation, we can reduce that time by 80 percent in large scale projects and free our artists to focus on other tasks."
Despite the huge advances in visual fidelity lately, there still aren't too many that can boast richly detailed and lifelike facial animations. Larger games such as RPGs suffer particularly in this regard, and if development time can be slashed 80%, we could potentially realistic facial animations throughout an entire game, even Grunt #543843.