ABSTRACT Generating human–object interaction (HOI) from text remains challenging because it must preserve both semantic alignment and human–object motion consistency. In practical animation synthesis and editing, partial motion observations are often available at the beginning and end of an interaction, making it more meaningful to synthesize the intermediate process under text guidance than to generate the entire sequence solely from text. We propose Blueprint2Motion, a two‐stage generative framework for HOI motion synthesis conditioned on text, object geometry, and spatiotemporal motion context. In the first stage, Text2Blueprint predicts an intermediate limb–object blueprint, including temporally coherent object motion and limb trajectories, from text, object geometry, and historical/future motion observations. In the second stage, Blueprint2Motion uses the generated blueprint as an explicit control signal to synthesize full‐body responsive human motion under the same text prompt. We further introduce an interaction loss to improve spatial alignment and motion coherence between key body joints and the manipulated object. Experiments on FullBodyManipulation and BEHAVE show improved overall motion quality and human–object coordination on most metrics, especially for manipulation‐oriented interactions. These results suggest that structured limb–object blueprints are an effective intermediate representation for text‐guided HOI synthesis under partial motion conditions.

Ecological validity remains essential for generalizing scientific research into real-world applications. However, current methods for crowd emotion detection lack ecological validity due to limited diversity samples in datasets. This paper proposes a crowd emotion detection framework that improves the ecological validity of models from both dataset and methodological perspectives. Firstly, we develop an Emotional Crowd Generator script within Grand Theft Auto V to generate a large-scale and diverse synthetic emotional crowd dataset, named Emotional-GTA (E-GTA). Secondly, we utilize prior features to enhance the model's generalization ability, especially for rare samples. Building on this, we introduce a dual-driven Graph-based Prior Feature and Image Fusion Network (GPIFN), which further strengthens our model's ecological validity from a methodological perspective. We propose a graphical representation that effectively constructs the Crowd Image Graph (CIG) and the Crowd Prior Features Graph (CPG). The CIG represents crowds from the perspective of the image features, while the CPG represents them from the perspective of prior features. We then design a dual-stream network GPIFN that extracts image features from the CIG and prior features from the CPG. Additionally, we design an Image and Prior Features Fusion Module (IPFM) that efficiently merges image and prior features while maintaining the original stream features. Our experiments demonstrate that both E-GTA and GPIFN greatly enhance ecological validity in real-world scenarios. Our framework achieves state-of-the-art results on real-world datasets: UMN and Violent-Flows.

The rapid advancement of deep learning and the exponential growth of multimodal data have led to increased attention on multimodal emotion analysis and comprehension in affect computing. While existing multimodal works have achieved notable results in emotion recognition, several challenges remain. First, the scarcity of public large-scale multimodal emotion datasets is attributed to the high cost of manual annotation and the subjectivity of handcrafted labels. Second, most approaches only focus on learning emotion category information, disregarding the crucial evaluation indicator of emotion intensity, which hampers the development of fine-grained emotion recognition. Third, a significant emotion semantic discrepancy exists in different modalities, and current methodologies struggle to bridge the cross-modal gap and effectively utilize a vast amount of unlabeled emotion data, hindering the production of high-quality pseudolabels and superior classification performance. To address these challenges, based on the multitask learning architecture, we propose a novel semisupervised fine-grained emotion recognition model SMEIR-net for multimodal emotion and intensity recognition. Concretely, in semisupervised learning (SSL) phase, we design multistage self-training and consistency regularization paradigm to generate high-quality pseudolabels. Then, in supervised learning phase, we leverage multimodal transformer fusion and adversarial learning to eliminate the cross-modal semantic discrepancy. Extensive experiments are conducted on three benchmark datasets, namely RAVDESS, eNTERFACE, and Lombard-GRID, to evaluate the proposed model. The series sets of experimental results demonstrate that our SSL model successfully utilizes multimodal data and available labels to transfer emotion and intensity information from labeled to unlabeled datasets. Moreover, the corresponding evaluation metrics demonstrate that the utilize high-quality pseudolabels can achieve superior emotion and intensity classification performance, which outperforms other state-of-the-art baselines under the same condition.

Abstract Crowd evacuation has gained increasing attention in recent years. The agent‐based method has shown a superior capability to simulate complex behaviors during crowd evacuation simulation. For agent modeling, most existing methods only consider the decision process but ignore the detailed physical motion. In this article, we propose a hierarchical framework for crowd evacuation simulation, which combines the agent decision model with the agent motion model. In the decision model, we integrate emotional contagion and scene information to determine global path planning and local collision avoidance. In the motion model, we introduce a physics‐based character control method and control agent motion using deep reinforcement learning. Based on the decision strategy, the decision model can use a signal to control the agent motion in the motion model. Compared with existing methods, our framework can simulate physical interactions between agents and the environment. The results of the crowd evacuation simulation demonstrate that our framework can simulate crowd evacuation with physical fidelity.

Safe and efficient navigation of a robot in a high-density and dynamic crowd is a challenging task. Most of existing navigation algorithms need to acquire the full dynamics of neighboring humans at all times, making them heavily dependent on a complex upper level information state estimation process. Moreover, the scene perception modules of existing algorithms do not comprehensively model human–robot interaction in the spatiotemporal dimension, leading to frequent freezing and collision problems in reinforcement learning-based navigation algorithms. To address the above problems, we propose a fine-grained spatiotemporal graphical attention navigation algorithm (FST-RL), which enriches the scene perception module of reinforcement learning algorithms by jointly encoding human–robot motion patterns, interhuman relations, and long-term dependent information of human–robot interactions. With the proposed algorithm, socially compatible navigation routes can be generated by the built-in spatiotemporal reasoning module with the premise of obtaining only the position information of agents in the robot’s perception domain. The experimental results show a significant improvement of FST-RL in terms of success rate (22.3% improvement), navigation time (13.6% reduction), and average return (20.8% improvement) in a high-density human environment compared with the current optimal navigation algorithm (DS-RNN). Ablation and qualitative experiments show that the scene perception module of FST-RL can effectively reduce the robot’s collision and conservative behaviors in challenging scenarios.

Empathy, an essential quality in daily human conversations, plays a crucial role in dialogue systems. In recent years, there has been a surge of interest among researchers in developing empathetic response generation. However, existing methods often ignore the high-level process of generating empathy-imagination, which involves consciously putting oneself in the other person's shoes during a conversation. To solve this critical problem, we propose a novel approach called EmpSOI that adopts self and other-imagine to generate empathetic responses. Specifically, we design two heterogeneous graphs to incorporate the two different imaginative perspectives: the self-imagine perspective and the other-imagine perspective, which enables the model to empathize with the user from different perspectives. Besides, we incorporate a gating mechanism to regulate the contribution of imagination information from these two perspectives during the response generation stage. The mechanism enables the model to differentiate between two imaginative perspectives. The results of extensive automatic and manual experiments illustrate the advantages of our model compared to other comparative models in perceiving the user's emotional state and generating empathetic responses.

Abstract An effective virtual agent can serve humans complete task‐based work efficaciously and manage interpersonal relationships with humans judiciously. This article investigates the effectiveness of emotional intelligence (EI) of a virtual agent taking over the role of a virtual tour guide (VTGuide) in a desktop application when witnessing a personal conflict between a human user and two virtual agents participating in the tour (the human user was ignored by the agents). A within‐subject experiment is conducted to verify the validity of EI. Participants rate VTGuides (with or without EI) on conflict handling and report their feelings during the interaction. In addition, objective behavioral data of users are recorded, including facial expressions and textual sentiment, to assess the perception of rapport. The results show that an emotionally intelligent VTGuide performs an agreeable behavior system (e.g., nodding, eye contact, friendly facial expressions), comforting verbal strategy (e.g., distracting attention, mediating between conflicting parties), and positive paralinguistic cues (e.g., smiling textual emojis). It can effectively mitigate intra‐group conflicts and maintain interpersonal relationships. Thus, demonstrating a stronger sense of EI can better transmit engagement, interest, understanding, and emotional feedback in complex relationships.

Abstract In most of fire evacuation, people follow the walls. The reasons behind this escape behavior have not been verified by experiments. In this article, we design a virtual reality (VR) fire evacuation game with realistic virtual environment to explore the effects of smoke concentration, individual familiarity with the environment, and neighbor behavior on individual wall‐following behavior. Individuals' familiarity with the environment is obtained through experimental training and the frequencies of their behavior along the wall at different smoke concentrations are recorded. From a subjective perspective, we use the presence scale to assess the immersion of the designed VR game. The results show that users have a good level of presence in the designed VR game. From an objective perspective, we analyze the game data using a statistically based approach. The analysis shows that high concentration of smoke, unfamiliar with the environment will increase individuals' reliance on walls, but neighbor behavior along the wall has no significant effect on individual wall‐following behavior.

In terrorist attack simulations, existing methods do not describe individual differences, which means different individuals will not have different behaviors. To address this problem, we propose a framework to model people’s heterogeneous behaviors in terrorist attack. For pedestrian, we construct an emotional model that takes into account its personality and visual perception. The emotional model is then combined with pedestrians' relationship networks to make the decision-making model. With the proposed decision-making model, pedestrian may have altruistic behaviors. For terrorist, a mapping model is developed to map its antisocial personality to its attacking strategy. The experiments show that the proposed algorithm can generate realistic heterogeneous behaviors that are consistent with existing psychological research findings.

Abstract During an earthquake, interior nonstructural components of a building will be damaged. The damaged objects will obstruct pedestrians' evacuation routes and increase casualties. But this issue has received scant attention in evacuation simulation research. This paper focuses on this issue and proposes an indoor seismic evacuation model to simulate crowd evacuation in a dynamic environment. A physical model of nonstructural components is presented to simulate the dynamics of the indoor scenario. The flow field algorithm is constructed to guide pedestrian's avoidance behaviors globally to reflect the impact of environmental changes on indoor crowd path selection, and a modified social force model is built to simulate the joint influence of seismic forces and the environment on pedestrian motion states. The results of the experiments shows that the proposed model can generate realistic evacuation scene and rational evacuation routes in the earthquake.

Abstract Since the differences of individual pedestrians and the diversity of pedestrian‐vehicle avoidance behaviors in real life, it is important to consider the behavioral heterogeneity of pedestrians and vehicles in simulation. Most existing simulation models cannot generate personalized pedestrian‐vehicle avoidance scenarios. Taking personality and emotional factors into account, we proposed a simulation method to realize various pedestrian‐vehicle collision avoidance scenarios by adjusting the parameter of personality trait. In this study, drivers' yielding strategies were classified as careful and aggressive based on their different personality traits. For pedestrians, in addition to personality traits, emotional factors were also introduced to achieve more realistic speed control. The experimental results showed that the proposed method could generate personalized pedestrian‐vehicle collision avoidance scenarios in multiple traffic scenarios.

Abstract In emergency situation, mass panic often causes more causalities than the disaster itself. The crowd emotional model could be used to simulate how crowd behavior in emergency scenarios and be helpful for developing crowd evacuation plans in emergency situations. However, existing crowd emotional models usually set model parameters in an empirical manner and are not validated by real cases. In this paper, a crowd emotional model is proposed to simulate the crowd movement in outdoor emergency situations. First of all, the crowd entropy and the movement difference are proposed to describe the emotional impact of the crowd scene on the agents. The perception of vision and hearing are considered, and the calculation formulas of the agent's emotional intensity and crowd emotional contagion are proposed. By calculating individual trajectories in the real video, the cumulative differences between the movements of the real crowd and the corresponding virtual crowd are analyzed. At last, a multi‐parameter optimization method is implemented by the differential evolution algorithm. To verify the parameters in models, three videos which are generated from three real cases, including explosion attack, shooting incident, and crowd disturbance are selected for experimental verification. The results showed that the proposed model could be a feasible method for optimizing parameters to simulate the emergency scenario.

Abstract With the increasing number of emergencies, the crowd simulation technology has attracted wide attention in the recent years. Existing emergencies have shown that individuals are easy to be influenced by others' emotion during the evacuation. This will make it easier for people to aggregate together and increase security risks. Some of the existing evacuation models without considering emotion are therefore not suitable for describing crowd behaviors in emergencies. We propose a perception‐based emotion contagion model and use multiagent technology to simulate crowd behaviors. Navigation points are introduced to guide the movement of the agents. Based on the proposed model, a prototype simulation system for crowd emotion contagion is developed. The comparative simulation experiments verify that the model can effectively deduct the evacuation time and crowd emotion contagion. The proposed model could be an assistant analysis method for crowd management in emergencies.