Head avatar reenactment focuses on creating animatable personal avatars from monocular videos, serving as a foundational element for applications like social signal understanding, gaming, human-machine interaction, and computer vision. Recent advances in 3D Morphable Model (3DMM)-based facial reconstruction methods have achieved remarkable high-fidelity face estimation. However, on the one hand, they struggle to capture the entire head, including non-facial regions and background details in real time, which is an essential aspect for producing realistic, high-fidelity head avatars. On the other hand, recent approaches leveraging generative adversarial networks (GANs) for head avatar generation from videos can achieve high-quality reenactments but encounter limitations in reproducing fine-grained head details, such as wrinkles and hair textures. In addition, existing methods generally rely on a large amount of training data, and rarely focus on using only a simple selfie video to achieve avatar reenactment. To address these challenges, this study introduces a method for detailed head avatar reenactment using a selfie video. The approach combines 3DMMs with a StyleGAN-based generator. A detailed reconstruction model is proposed, incorporating mixed loss functions for foreground reconstruction and avatar image generation during adversarial training to recover high-frequency details. Qualitative and quantitative evaluations on self-reenactment and cross-reenactment tasks demonstrate that the proposed method achieves superior head avatar reconstruction with rich and intricate textures compared to existing approaches.

Smiling faces are often preferred in daily social interactions. Many socially interactive human-like virtual agents are equipped with the capability to produce standardized smiles that are widely considered to be universal. However, mounting evidence shows that people from different cultures prefer different smiles. To engage a culturally diverse range of human users, socially interactive human-like virtual agents must be equipped with culturally-valid dynamic facial expressions. To develop culturally sensitive smiles, we use data-driven, perception-based methods to model the facial expressions of happy in 60 individuals in two distinct cultures (East Asian and Western European). On each experimental trial, we generated a random facial animation composed of a random sub-set of individual face movements (i.e., AUs), each with a random movement. Each cultural participant categorized 2400 such facial animations according to an emotion label (e.g., happy) if appropriate, otherwise selecting ‘other.’ We derived facial expression models of happy for each cultural participant by measuring the statistical relationship between the dynamic AUs presented on each trial and each participant's responses. Analysis of the facial expression models revealed clear cross-cultural similarity and diversity in smiles–for example, smiling with raised cheeks (AU12-6) is culturally common, while open-mouth smiling (AU25-12) is Western-specific and smiling with eyebrow raising (AU1-2) is East Asian-specific. Analysis of the temporal dynamics of each AU further revealed cultural diversity in smiles. We anticipate that our approach will improve the social signalling capabilities of socially interactive human-like virtual agents and broaden their usability in global market.

Measuring facial expressions is a notoriously difficult and time-consuming process, often involving manual labeling of low-dimensional descriptors such as Action Units (AUs). Computer vision algorithms provide automated alternatives for measuring and annotating face shape and expression from 2D images, but often ignore the complexities of dynamic 3D facial expressions. Moreover, open-source implementations are often difficult to use, preventing widespread adoption by the wider scientific community beyond computer vision. To address these issues, we develop dynamic face imaging, a novel analysis framework to study social face perception and expression. We use state-of-the-art 3D face reconstruction models to quantify face movement as temporal shape deviations in a common 3D mesh topology, which disentangles global (head) movement and local (facial) movement. Using a set of validation analyses, we test different reconstruction algorithms and quantify how well they reconstruct facial “action units” and track key facial landmarks in 3D, demonstrating promising performance and highlight areas for improvement. We provide an open-source software package that implements functionality for easy reconstruction, preprocessing, and analysis of these dynamic facial expression data.

Facial expressions are used to facilitate daily interactions including conversations, in every culture. However, little is known about which specific facial expressions convey conversational messages, which limits the social signalling capabilities of conversational virtual agents. We address this critical knowledge gap by modeling the facial expressions of key conversational messages directly from cultural perception, and transferring them to a conversational virtual agent. Using a novel data-driven psychology-based method, we modelled facial expressions of 'thinking,' 'interested,' 'bored' and 'confused' directly from the cultural perception of 40 participants across two distinct cultures (Western European, East Asian). We then transferred these facial expression models to a popular conversational virtual agent and validated them with a new group of cultural participants. Results showed that, in both cultures, the majority of our culturally derived conversational facial expression models successfully transferred to the agent. A further cross-cultural analysis of the conversational facial expressions showed systematic similarities (e.g., eye brow raising in 'interested') and differences (e.g., Westerners use horizontal mouth stretch and East Asians use vertical mouth opening to show confusion) that could facilitate or hinder cross-cultural communication. Our results demonstrate the power of using a culturally sensitive perception-based psychological approach to develop psychologically impactful facial expressions for conversational virtual agents. We anticipate that our facial expression models will enhance virtual agent social signalling capabilities and their global marketability.

Most socially interactive virtual agents that generate facial expressions lack critical visual features such as expressive wrinkles, which could reduce their realistic appearance. Here, we examined the impact of dynamic facial texture on perceptions of realism of facial expressions of emotion and identified the emotion-specific features that enhance this perception. In a human perceptual judgment task, participants (20 white Westerners, 10 female) viewed pairs of facial expressions of the six classic emotions - happy, surprise, fear, disgust, anger and sad - with and without dynamic textures and selected the most realistic one from the pair. Analysis of participant choices showed that facial expressions with dynamic texture are perceived significantly more often as more realistic for all emotions except sad. Further analysis of the facial expression signals showed that emotion-specific features, such as darker forehead furrows in surprise, unilateral nose wrinkling in disgust, and shade variations around the cheeks in happy, enhanced perceptions of realism. Together, our results highlight the importance of equipping virtual agents with dynamic face movement texture to produce realistic facial expressions of emotion.

Virtual agents with social characteristics such as warmth and competence, which are perceived readily from the face, impact human behavior effectively. However, generating such social signals is challenging because faces are complex, high-dimensional information spaces of 3D shape and complexion. We address this challenge using a data-driven psychology-based method that can objectively model socially impactful face features from subjective human perception. Specifically, we modeled the 3D shape and 2D complexion face features that drive four key social perceptions - competence, dominance, warmth, and trustworthiness. Analysis of the face models revealed that the four key traits are systematically structured by a set of latent face features including eyebrow height and chin protrusion, which can thus be used to efficiently generate social trait face features in virtual agents. We anticipate that our psychologically valid generative model of social face signals will enhance the social signaling capabilities of virtual agents.

To engage their human users, socially interactive virtual agents must be equipped with the ability to communicate emotions using facial expressions. Therefore, a main goal is to build a generative model that can produce the range of realistic dynamic facial expressions of emotion that occur across social life. We contribute to this goal by building a psychologically valid generative model of facial expressions directly from subjective human perception using a novel psychology-based approach. First, we build a valence-arousal space of face movements by identifying the specific face movements that convey valence (positive/negative) and arousal (excited/calm) in 40 individual participants. We then tested the capacity of the valence-arousal space to generate a broad range of facial expressions of emotion including the six classic emotions and complex emotions. By cross-correlating a large set of facial expressions of basic and complex emotions with the valence-arousal space, we show that our model can successfully represent a wide range of emotions. We anticipate that our psychologically valid facial expression generation model will enhance the emotion signalling capabilities of virtual agents.

Equipping virtual agents with attractive face can facilitate human-machine interactions. Given the real-world diversity of human's aesthetic preference and the complexity of 3D face information that comprises high-dimensional shape and complexion variations, synthesizing attractive faces for virtual human is particularly challenging. Here, we addressed this challenge using a psychological data-driven approach that reverse engineers the attractive features perceived by human participants from the randomly generated faces that capture the nature distribution of multivariate 3D shape and 2D complexion variations in human population, separately for Western European and East Asian cultures. Our analysis revealed a set of features that can increase the attractiveness of faces in each culture, including their cultural idiosyncrasies. These human and culturally validated features can thus be synthesized on virtual agents to enhance their utility in practical applications.

Social robots must be able to generate realistic and recognizable facial expressions to engage their human users. Many social robots are equipped with standardized facial expressions of emotion that are widely considered to be universally recognized across all cultures. However, mounting evidence shows that these facial expressions are not universally recognized - for example, they elicit significantly lower recognition accuracy in East Asian cultures than they do in Western cultures. Therefore, without culturally sensitive facial expressions, state-of-the-art social robots are restricted in their ability to engage a culturally diverse range of human users, which in turn limits their global marketability. To develop culturally sensitive facial expressions, novel data-driven methods are used to model the dynamic face movement patterns that convey basic emotions (e.g., happy, sad, anger) in a given culture using cultural perception. Here, we tested whether such dynamic facial expression models, derived in an East Asian culture and transferred to a popular social robot, improved the social signalling generation capabilities of the social robot with East Asian participants. Results showed that, compared to the social robot's existing set of facial `universal' expressions, the culturally-sensitive facial expression models are recognized with generally higher accuracy and judged as more human-like by East Asian participants. We also detail the specific dynamic face movements (Action Units) that are associated with high recognition accuracy and judgments of human-likeness, including those that further boost performance. Our results therefore demonstrate the utility of using data-driven methods that employ human cultural perception to derive culturally-sensitive facial expressions that improve the social face signal generation capabilities of social robots. We anticipate that these methods will continue to inform the design of social robots and broaden their usability and global marketability.

Social robots are now part of human society, destined for schools, hospitals, and homes to perform a variety of tasks. To engage their human users, social robots must be equipped with the essential social skill of facial expression communication. Yet, even state-of-the-art social robots are limited in this ability because they often rely on a restricted set of facial expressions derived from theory with well-known limitations such as lacking naturalistic dynamics. With no agreed methodology to objectively engineer a broader variance of more psychologically impactful facial expressions into the social robots' repertoire, human-robot interactions remain restricted. Here, we address this generic challenge with new methodologies that can reverse-engineer dynamic facial expressions into a social robot head. Our data-driven, user-centered approach, which combines human perception with psychophysical methods, produced highly recognizable and human-like dynamic facial expressions of the six classic emotions that generally outperformed state-of-art social robot facial expressions. Our data demonstrates the feasibility of our method applied to social robotics and highlights the benefits of using a data-driven approach that puts human users as central to deriving facial expressions for social robots. We also discuss future work to reverse-engineer a wider range of socially relevant facial expressions including conversational messages (e.g., interest, confusion) and personality traits (e.g., trustworthiness, attractiveness). Together, our results highlight the key role that psychology must continue to play in the design of social robots.

To understand visual cognition, it is imperative to determine when, how and with what information the human brain categorizes the visual input. Visual categorization consistently involves at least an early and a late stage: the occipito-temporal N170 event related potential related to stimulus encoding and the parietal P300 involved in perceptual decisions. Here we sought to understand how the brain globally transforms its representations of face categories from their early encoding to the later decision stage over the 400 ms time window encompassing the N170 and P300 brain events. We applied classification image techniques to the behavioral and electroencephalographic data of three observers who categorized seven facial expressions of emotion and report two main findings: (1) over the 400 ms time course, processing of facial features initially spreads bilaterally across the left and right occipito-temporal regions to dynamically converge onto the centro-parietal region; (2) concurrently, information processing gradually shifts from encoding common face features across all spatial scales (e.g., the eyes) to representing only the finer scales of the diagnostic features that are richer in useful information for behavior (e.g., the wide opened eyes in 'fear'; the detailed mouth in 'happy'). Our findings suggest that the brain refines its diagnostic representations of visual categories over the first 400 ms of processing by trimming a thorough encoding of features over the N170, to leave only the detailed information important for perceptual decisions over the P300.

The brain computations that underlie our ability to recognize facial expressions involve the extraction of relevant information from the faces of our peers, and allow us to readily respond in an appropriate manner to the displayed emotion. Here we present recent advances in understanding the brain processes underlying the categorization of facial expressions of emotion using classification image techniques to link both the brain dynamics (EEG) and the behavioural strategies of three observers with specific facial features.

We present hybrid images , a technique that produces static images with two interpretations, which change as a function of viewing distance. Hybrid images are based on the multiscale processing of images by the human visual system and are motivated by masking studies in visual perception. These images can be used to create compelling displays in which the image appears to change as the viewing distance changes. We show that by taking into account perceptual grouping mechanisms it is possible to build compelling hybrid images with stable percepts at each distance. We show examples in which hybrid images are used to create textures that become visible only when seen up-close, to generate facial expressions whose interpretation changes with viewing distance, and to visualize changes over time within a single picture.