Speech-driven 3D facial animation aims to generate realistic facial meshes based on input speech signals. However, due to a lack of understanding of visible articulatory movements, current state-of-the-art methods result in inaccurate lip and jaw movements. Traditional evaluation metrics, such as lip vertex error (LVE), often fail to represent the quality of visual results. Based on our observation, we reveal the problems with existing evaluation metrics and raise the necessity for separate evaluation approaches for 3D axes. Comprehensive analysis shows that most recent methods struggle to precisely predict lip and jaw movements in 3D space.

In this work, we propose methods for automatically generating primitive skills from demonstration of a task. Additionally, we propose methods for improving the existing primitive skills and adding the new primitive skills incrementally and automatically. To validate our proposed methods, we present experimental results of a human-like robot handling three gestures and a task for making coffee.