Nonverbal communication plays a crucial role in both human-human and human-robot interactions (HRIs), where facial expressions convey emotions, intentions and trust. Enabling humanoid robots to generate human-like facial reactions in response to human speech and facial behaviours remains significant challenges. In this work, we leverage human-human interaction (HHI) datasets to train a humanoid robot, allowing it to learn and imitate facial reactions to both speech and facial expression inputs. Specifically, we extend a sequence-to-sequence (Seq2Seq)-based framework that enables robots to simulate human-like virtual facial expressions that are appropriate for responding to the perceived human user behaviours. Then, we propose a deep neural network-based motor mapping model to translate these expressions into physical robot movements. Experiments demonstrate that our facial reaction–motor mapping framework successfully enables robotic self-reactions to various human behaviours, where our model can best predict 50 frames (two seconds) of facial reactions in response to the input user behaviour of the same duration, aligning with human cognitive and neuromuscular processes. Our code is provided at https://github.com/mrsgzg/Robot_Face_Reaction.

The seamless transition of control between drivers and autonomous systems remains a critical challenge in automated driving, affecting both safety outcomes and overall user experience. To address this challenge, our study examines the effectiveness of two distinct haptic feedback approaches—pneumatic and vibrotactile—when implemented as intelligent interface components for takeover requests (TORs) during these transition periods. We specifically investigate how these haptic modalities can effectively signal drivers when human intervention is required, facilitating smoother control transitions from automated to manual driving. We designed a comprehensive experimental setup integrating these haptic modalities with audio and visual cues and evaluated their performance across nine interaction tasks to understand how multi-modal feedback influences driver responsiveness during takeover scenarios. Our findings reveal that multi-modal approaches incorporating either pneumatic or vibrotactile feedback, combined with standard visual cues, substantially outperform audio-only alerts in both response time and accuracy metrics for takeover requests (TORs). Notably, pneumatic feedback offered more natural sensation and smoother transitions than vibrotactile feedback, with pneumatic systems excelling in comfort while vibrotactile feedback better serves urgent takeovers. This first systematic comparison provides valuable insights for developing interfaces that balance effectiveness with comfort in human-machine systems.

In this paper, we have documented the challenges that drivers with autopilots experience on real-world roads, by focusing on the practices of humans taking over. We analyze data of full self-driving cars selected from third-party YouTube videos in a conversation analytic approach. We have shown how drivers treat the car’s moment-by-moment motion as actions that are projectable for potentially relevant risky outcomes, and how they take over the full self-driving system in situ and in vivo, with continuous situated monitoring. We have demonstrated four typical situations in which drivers take over in the unfolding course of driving action, that is, going too close to the front car, inappropriate speed in the local context, wrong recognition of lanes, and pedestrian priority. We argue that the achievement of human takeovers is inextricably connected to the situated organization and accountability of the course of action.

Live streaming with mobile phones is a common practice where streamers and viewers use various resources for interaction. Based on the method of multimodal conversation analysis, we examine recordings of a clay sculptor's live streams on a Chinese social app. We address how the streamer's dual involvements—doing the sculpture work and responding to viewers’ messages—are achieved moment-by-moment. We will demonstrate how the streamer uses multiple resources, such as language, body torque, facial expressions, eye gaze, phone adjustment, and the “disrupted turn adjacency” feature of viewers’ messages, to achieve multiactivity by holding two intersecting courses of action, and how he may use live streaming to achieve self-exposure, chat, and virtual intimacy during his routine work.

To develop robots that can show cognitive functions, we must learn from the knowledge of human cognition. Existing biological and psychological evidence suggests that self-face perception and sensorimotor learning mechanisms play a crucial role in self-recognition. However, one of the most important self-identity cues – facial information – has not been extensively studied in the robot self-recognition task. Current research on robot self-recognition primarily relies on the recognition of high-precision targets and tracking of manipulator motions, where the self-perception of facial information is not well studied. In this work, we propose a novel approach to achieve self-recognition via self-perception of facial expressions. Specifically, we developed a Conditional Generative Adversarial Network (CGAN) model using the knowledge on human cognitive and sensorimotor functions. It allows the robot to be aware of self-face (i.e., off-line model). Passing the observed visual variations in a mirror and comparing them to self-perceptive information, the robot can recognize the self through an online Bayesian learning regression. The results of our first experiment show that the robot can recognize itself in a mirror. The results from the second experiment show that our algorithm could be tricked by a similar robot with the same facial expressions, which is similar to the rubber hand illusion (RHI).