Emotion motivates behavior. Investigating the correlation between behavior and emotion, an often overlooked perspective, plays a significant role in uncovering the underlying motives behind behaviors and the intrinsic cause-effects of social events. This article proposes a methodology for mining the correlation between public behavior and emotion using daily news data. Initially, aspect-emotion-reaction (A-E-R) triplets are extracted and generalized, encompassing both explicit and implicit patterns. Then, a knowledge representation model based on hypothetical context (KRHC) with a self-reflection mechanism is proposed to uncover implicit relationships between emotion and behavior through attention mechanisms. By combining rule-based methods for explicit relationships and deep learning for implicit ones, an understanding of emotion-behavior patterns is achieved. In this study, the behaviors are divided into three categories of prosocial, antisocial, and normal behaviors with ten secondary types. Seven categories of emotions are adopted. The proposed deep learning model KRHC is validated on A-E-R datasets and public KINSHIP datasets. The experiment results are concluded; for example, when "fear," "sad," and "surprise" emotions appear, it drives behavior "panic" with most probability. These findings could provide insights for both human-computer interaction and public safety management applications.

While significant progress has been made in recognizing entities from plain text, the exploration of entity recognition from multimodal data remains limited due to disparities in semantic representation. In light of this challenge, given the supportive nature of visual and text data, we propose a novel entity recognition model called Heterogeneous Graph Reasoning(HGR), leveraging the synergistic nature of visual and textual data. This is achieved through the utilization of the Vision Refine and Graph Cross Inference modules. In the Vision Refine module, semantically relevant objects hidden in the image are selected to aid in the text entity extraction. In the Graph Cross Inference module, cross-association inference between visual regions and textual entities is constructed through graph construction, heterogeneous graph fusion, visual region refinement and cross inference. Extensive experiments on four multimodal datasets are demonstrate the superiority of our model, when compared to the second-best state-of-the-art model.

Multimodal emotion recognition comprehensively identifies the emotion contained in multimodal data by bridging the gaps between heterogeneous dataset. In recent years, multimodal emotion recognition methods have gained significant attention and been shown to surpass single-modal approaches. Most of the existing multimodal emotion analysis methods simply combine different modalities to improve the recognition capability of consistent emotion expressions across multimodal data. However, it remains challenging to recognize the right emotion when multiple modalities' contents carry inconsistent or even contradictory emotions. To solve this problem, we propose a novel image-text emotion recognition model named Multimodal Cross-Attention Bayesian Network(MCABN). The entire network exploits Bayesian theory to learn the distribution of its weight parameters, making optimization directions and results of parameters interpretable. What's more, the model leverages the consistency and complementarity between visual content and textual description to arrive at accurate decisions. Specifically, for each modality, multiple explainable features (color, texture, and shape feature in the image, while adjective, adverb, verb, noun, and negative feature in the text) and one unexplainable feature are fused as its feature representation to reinforce emotion-related features. Then two single-modal attention modules(Visual Attention Module and Textual Attention Module) capture the most discriminative features in a single image and text; Two cross-modal attention modules(Image-guided Text Attention Module and Text-guided Image Attention Module) extract the complementary and dominant features between two modalities by interactive learning. Finally, the outputs of four attention modules are integrated through intermediate fusion to predict the final emotion. The experimental results on the NVTD and MVSA-Multiple dataset indicate that the proposed MCABN outperforms state-of-the-art baselines by substantial margins.