Nvidia and Remedy Using Neural Networks for Lifelike Facial Animation in Record Time

Written by Stuart Thomas on Thu, Aug 3, 2017 9:51 AM

Remedy’s not usually one of the first names you associate with top-notch facial animation. These were the culprits behind Max Payne’s perma-strained face, looking for all the world like he’d just prolapsed. Still, Alan Wake and Quantum Break were alright. Still seeking to atone for Max’s unfortunate expression, Remedy has teamed up with Nvidia to work on dramatically improving motion capture and facial animations using deep convolutional neural networks.

Their new system drastically reduces the amount of time and effort needed to create lifelike facial animations for characters, converting real footage of a person into gameplay with 5-10 minutes.

Remedy and Nvidia have achieved this by creating a real-time deep learning framework for facial performance capture, firstly capturing footage using multi-view tracking to provide a 3D image. A convolutional neural network then trains itself to identify self-occluded regions and then produces a high-quality output render. Perhaps most impressive of all, once the neural network has a detailed profile of an actor, it can even render realistic facial animation using an audio clip alone.

What once took hours could now take minutes, which would be a major boon in delivering lifelike facial animations to characters. With the current slow process, it can be too much of a time sink for all but the largest devs, and this advanced technology could speed up the process dramatically.

"Based on the Nvidia Research work we've seen in AI-driven facial animation, we're convinced AI will revolutionise content creation," said Antti Herva, lead character technical artist at Remedy. "Complex facial animation for digital doubles like that in Quantum Break can take several man-years to create. After working with Nvidia to build video- and audio-driven deep neural networks for facial animation, we can reduce that time by 80 percent in large scale projects and free our artists to focus on other tasks."

Despite the huge advances in visual fidelity lately, there still aren't too many that can boast richly detailed and lifelike facial animations. Larger games such as RPGs suffer particularly in this regard, and if development time can be slashed 80%, we could potentially realistic facial animations throughout an entire game, even Grunt #543843.

Login or Register to join the debate

Rep
-13
Offline
17:12 Aug-03-2017

use this on next mass effect please ea

9
Rep
38
Offline
14:21 Aug-03-2017

Rockstar, please use this technology and release GTA 6 in 2018

6
Rep
132
Offline
12:14 Aug-03-2017

This is pretty awesome.

2
Rep
3
Offline
11:37 Aug-03-2017

If only Bioware teamed up and used this

4
Rep
-67
Offline
11:39 Aug-03-2017

Trust me, in a near future this will be the standard.

1
Rep
191
Offline
junior admin badge
18:35 Aug-03-2017

Wouldn't surprise me 1 bit.

0
Rep
191
Offline
junior admin badge
10:42 Aug-03-2017

Now if this doesn't speed up AAA game releases nothing will.

0
Rep
-67
Offline
10:10 Aug-03-2017

Neural Networks for srcnn (image upscaling), image interpolation, artificial intelligence and now animations. The future is truly Neural Networks. However, we need FAAAAAAAAASTER cpu's!!

2
Rep
191
Offline
junior admin badge
10:42 Aug-03-2017

Or gpu-s. ;)

0
Rep
-67
Offline
11:16 Aug-03-2017

Well, actually both. I think we need tech 15 - 20 times faster before we are going to see this coming to real good usage.

2
Rep
191
Offline
junior admin badge
11:23 Aug-03-2017

With 16 times stronger GPU-s you will no longer be able to differentiate between VR and RW. (RW Real World).

0

Can They Run... |

| 30FPS, Low, 1080p
Core i7-3770 4-Core 3.4GHz GeForce GTX 1650 MSI Ventus XS OC 4GB 16GB
| 60FPS, Medium, 1080p
Core i3-9100 4-Core 3.6GHz GeForce GTX 1050 Ti 4GB 8GB
| 60FPS, High, 1080p
Core i5-3470 3.2GHz Radeon RX 470 Sapphire Nitro+ 8GB 16GB
| 60FPS, High, 1080p
Core i5-10300H 4-Core 2.50GHz GeForce GTX 1050 Ti 4GB 16GB
100% Yes [1 votes]
| 60FPS, High, 1080p
Core i3-1005G1 2-Core 1.20GHz UHD Graphics 630 4GB
| 60FPS, Medium, 720p
Core i3-1005G1 2-Core 1.20GHz UHD Graphics 630 4GB