Performance Driven Animated Faces

Here are some notes gathered to understand how to understand and create facial animations for the ARENA with blendshapes and other model formats.

Overview

• Practice and Theory of Blendshape Facial Models [1]. It’s a good starting point, with a nice overview of how blendshape works.

Blendshapes

• Blend Shape Interpolation and FACS for Realistic Avatar [2] (Facial Action Coding)
• Face Scan App (developers)
• Face Cap App (product)

Exporting

How do we get models into the ARENA?

1. Models are ASCII FBX with text file for the rigging sequence.
2. Convert ASCII FBX -> Binary FBX with: https://www.autodesk.com/developer-network/platform-technologies/fbx-converter-archives.
3. AutoDesk FAILED?! Use this instead: https://overbits.herokuapp.com/fbxgltf/.
4. Convert Binary FBX to GLTF with blendshape animations: https://github.com/facebookincubator/FBX2glTF.

Retargeting

How do we go from landmarks to expressions?

• Feature points based facial animation retargeting [3]
• Facial Landmark Detection: A Literature Survey [4]

FLAME

• Faces Learned with an Articulated Model and Expressions: https://flame.is.tue.mpg.de/
• Early Hao Li project on using ML to do face rigging.
• Very cool web demo: https://flame.is.tue.mpg.de/interactivemodelviewer.html

References

[1] Lewis, J.P., Anjyo, K., Rhee, T., Zhang, M., Pighin, F.H., & Deng, Z. (2014). Practice and Theory of Blendshape Facial Models. Eurographics.

[2] Alkawaz, M.H., Mohamad, D.B., Basori, A.H., & Saba, T. (2015). Blend Shape Interpolation and FACS for Realistic Avatar. 3D Research, 6, 1-10.

[3] Dutreve, L., Meyer, A., & Bouakaz, S. (2008). Feature points based facial animation retargeting. Virtual Reality Software and Technology.

[4] Wu, Y., & Ji, Q. (2018). Facial Landmark Detection: A Literature Survey. International Journal of Computer Vision, 127, 115-142.

---