The ability to handle multi-view facial expressions is important for computers to understand affective behavior under less constrained environment. However, most of existing methods for facial expression recognition are based on the near-frontal view face data, which are likely to fail in the non-frontal facial expression analysis. In this paper, we conduct an investigation on analyzing multi-view facial expressions. Three local patch descriptors (HoG, LBP, and SIFT) are used to extract facial features, which are the inputs to a nearest-neighbor indexing method that identifies facial expressions. We also investigate the influence of feature dimension reductions (PCA, LDA, and LPP) and classifier fusion on the recognition performance. We test our approaches on multi-view data generated from BU-3DFE 3D facial expression database that includes 100 subjects with 6 emotions and 4 intensity levels. Our extensive person-independent experiments suggest that the SIFT descriptor outperforms HoG and LBP, and LPP outperforms PCA and LDA in this application. But the classifier fusion does not show a significant advantage over SIFT-only classifier.

The existing methods of facial expression recognition are typically based on the near-frontal face data. The analysis of non-frontal-view facial expression is a largely unexplored research. The accessibility to a recent 3D facial expression database (BU-3DFE database) motivates us to explore an interesting question: whether non-frontal-view facial expression analysis can achieve the same as or better performance than the existing frontal-view facial expression method. Our extensive recognition experiments on data of 100 subjects with 5 yaw rotation view angles suggests that the non-frontal-view facial expression classification can outperform frontal-view facial expression recognition, given the manually labeled facial key points.

The creation of facial range models by 3D imaging systems has led to extensive work on 3D face recognition [19]. However, little work has been done to study the usefulness of such data for recognizing and understanding facial expressions. Psychological research shows that the shape of a human face, a highly mobile facial surface, is critical to facial expression perception. In this paper, we investigate the importance and usefulness of 3D facial geometric shapes to represent and recognize facial expressions using 3D facial expression range data. We propose a novel approach to extract primitive 3D facial expression features, and then apply the feature distribution to classify the prototypic facial expressions. In order to validate our proposed approach, we have conducted experiments for person-independent facial expression recognition using our newly created 3D facial expression database. We also demonstrate the advantages of our 3D geometric based approach over 2D texture based approaches in terms of various head poses.

Advanced human computer interaction requires automatic reading of human face in order to make the computer interact with human in the same way as human-to-human communication. We developed an automatic face tracking and lip reading system through a 3D face avatar to facilitate HCI applications in speech learning, emotional state monitoring, and non-verbal human computer interface design. The system implements a novel active face feature tracking algorithm with an uncalibrated camera. The 3D face pose is estimated and tracked by a Kalman filter-based matching process with a dynamic face model updating and constraint. The obtained facial motion parameters are transferred to an individualized 3D face avatar. As a result, a person's lip shape or expressions can be cloned to the animated 3D face avatar, by which all lip shapes from the same speech of different subjects can be easily compared and measured. This real time system targets the automatic facial expression analysis and synthesis for the next generation of HCI design.