In this paper an interaction system based on a three dimensional virtual head projected onto a pyramidal display is proposed. The proposed system makes use of a social robot behavioral architecture already developed in our lab, which allows us to interchange developments between our robotic realizations and the 3D avatar. The overall system is divided into two parts: back projection subsystem and expression generator subsystem. The back projection subsystem projects a three-dimensional avatar onto a pyramidal structure in order to achieve a sensation of depth and realism. The expression generator subsystem carries out the avatar animations using shape keys and bones, following the Facial Action Coding System (FACS). The system consists in several nodes that are integrated in ROS middleware (Robotic Operating System), and includes a user interface that makes the avatar teleoperation easier (the package is avaible in github public respository). In order to evaluate the expressiveness of the system, two sets of experiments have been performed: one to analyze the avatar's gestural ability, that is, its capability to perform expressions that can be identified by an observer, and a second experiment to measure the emotion displaying ability in terms of valence and arousal.

As part of a multimodal animated avatar previously presented in Marcos-Pablos et al. ((2010) A realistic, virtual head for human-computer interaction. Interact. Comput., 22, 176–192, ISSN 0953-5438), in this paper we describe a method for dynamic recognition of displayed facial emotions on low-resolution streaming images. First, we address the detection of action units (AUs) of the facial action coding system using active shape models and Gabor filters. Normalized outputs of the AU recognition step are then used as inputs for a neural network that consists of an habituation network plus a competitive network. Both the competitive and the habituation layer use differential equations, thus taking into account the dynamic information of facial expressions through time. Experimental results carried out on live video sequences and on the Cohn-Kanade face database show that the proposed method provides high recognition hit rates. To assess the suitability of the developed emotional recognition system for human–computer interaction applications, it has been successfully integrated in the architecture of an avatar and we have conducted a preliminary experiment on empathy. The experiment showed promising results, as the avatar that made use of the emotional recognition system obtained a clear increase in the positivity of the rating when compared with the same avatar with no emotional response.

Based on our previous work in the development of a multimodal animated avatar oriented to human-machine interaction and social robotics, a facial expression recognition system is presented in this paper. Both the avatar's architecture and facial recognition system have been oriented to FACS Action Units recognition and generation, so they can be easily integrated. The resulting animated system is able to maintain nonverbal and verbal bidirectional communication, enhancing the interaction with the avatar and making it more natural for the users.

In this paper an interactive and realistic virtual head oriented to human–computer interaction and social robotics is presented. It has been designed following a hybrid approach, taking robotic characteristics into account and searching for a convergence between these characteristics, real facial actions and animation techniques. An initial head model is first obtained from a real person using a laser scanner. Then the model is animated using a hierarchical skeleton based procedure. The proposed rig structure is close to real facial muscular anatomy and its behaviour follows the Facial Action Coding System. Speech synthesis and visual human-face tracking capabilities are also integrated for providing the head with further interaction ability. Using the said hybrid approach, the head can be readily linked to a social-robot architecture. The opinions of a number of persons interacting with this social avatar have been evaluated and are reported in the paper, as against their reactions when interacting with a social robot with a mechatronic face. Results show the suitability of the avatar for on-screen, real-time interfacing in human–computer interaction. The proposed technique could also be helpful in the future for designing and parameterizing mechatronic human-like heads for social robots.

In this paper an interactive and realistic facial animation oriented to human-computer interaction and social robotics is proposed. An initial model is obtained from a real face using a range scanner, and is then animated using a hierarchical skeleton oriented procedure. The rig structure is close to that of real facial muscular anatomy and the behavior follows the Facial Action Coding System. The resulting animated face is suitable for both real-time interfacing in human-machine interaction such as a social robot, and for helping in design and parametrization of android heads.