Enabling photorealistic avatar animations in virtual and augmented reality (VR/AR) has been challenging because of the difficulty of obtaining ground truth state of faces. It is physically impossible to obtain synchronized images from head-mounted cameras (HMC) sensing input, which has partial observations in infrared (IR), and an array of outside-in dome cameras, which have full observations that match avatars' appearance. Prior works relying on analysis-by-synthesis methods could generate accurate ground truth, but suffer from imperfect disentanglement between expression and style in their personalized training. The reliance of extensive paired captures (HMC and dome) for the same subject makes it operationally expensive to collect large-scale datasets, which cannot be reused for different HMC viewpoints and lighting. In this work, we propose a novel generative approach, Generative HMC (GenHMC), that leverages large unpaired HMC captures , which are much easier to collect, to directly generate high-quality synthetic HMC images given any conditioning avatar state from dome captures. We show that our method is able to properly disentangle the input conditioning signal that specifies facial expression and viewpoint, from facial appearance, leading to more accurate ground truth. Furthermore, our method can generalize to unseen identities, removing the reliance on the paired captures. We demonstrate these breakthroughs by both evaluating synthetic HMC images and universal face encoders trained from these new HMC-avatar correspondences, which achieve better data efficiency and state-of-the-art accuracy.

Human emotion recognition has been widely used and intensively studied in many areas such as e-Commerce, online education, healthcare, human-computer interaction and recently human-centric software engineering (HCSE). HCSE investigates human factors in the entire software development lifecycle, and human emotion can be used in many scenarios such as requirement gathering and usability testing. However, even though existing studies have already shown the advantages of emotion recognition with multi-source data such as text, audio and video, the current research and practice in HCSE are primarily based on single-source data. In addition, emotion recognition in HCSE faces several challenges such as multiple participants, changing environments and real-time requirement. To tackle these challenges, this paper proposes Edge4Emotion, a novel edge computing-based multisource human emotion recognition platform for HCSE. Edge4Emotion takes the advantage of the edge computing paradigm to efficiently support the collection of multi-source data such as audio, video and physiological data from various IoT devices, and emotion recognition with both single- and multisource models. As an on-going project, this paper focuses on the platform design and the preliminary evaluation of the platform with representative emotion recognition applications. The platform will be further extended to include more multi-source learning models and serve as an open-source platform for the development and evaluation of multi-source emotion recognition models for HCSE.

Recognition of human behaviours including body motions and facial expressions plays a significant role in human-centric software engineering. However, due to the data and computation intensive nature of human behaviour recognition through video analytics, expensive powerful machines are often required, which could hinder the research and application in human-centric software engineering. To address such an issue, this paper proposes a cost-effective human behaviour recognition system named Edge4Real which can be easily deployed in an edge computing environment with commodity machines. Compared with existing centralised solutions, Edge4Real has three major advantages including cost-effectiveness, easy-to-use, and real-time. Specifically, Edge4Real adopts a distributed architecture where components such as motion capturing, human behaviour recognition, data decoding and extraction, and the application of the recognition result, can be deployed on separated end devices and edge nodes in an edge computing environment. Using a virtual reality application which can capture a user's motion and translate into the motion of a 3D avatar in real time, we successfully validate the effectiveness of the system and demonstrate its promising value to the research and application of human-centric software engineering. The demo video can be found at https://youtu.be/tnEshD8j-kA.