Multimodal fusion aims to integrate complementary information from different modalities, yet it is often hindered by data distribution imbalance and cross-modal conflicts. In this paper, we propose Beneficial Noise Learning for Robust Multimodal Fusion (BRMF), a novel framework that reframes these challenges as beneficial perturbations signals for robust training. Specifically, BRMF first learns a stable and consistent joint representation within a Gaussian latent space under distributional constraints. Based on this joint representation, it then constructs diverse distributional environments and further encourages invariance by extending invariant risk minimization (IRM) to multimodal settings. Extensive experiments across multiple benchmarks demonstrate that BRMF exhibits superior robustness to noise and establishes new state-of-the-art performance.

Multimodal Sentiment Analysis (MSA) with incomplete data has gained significant attention recently.Existing studies focus on optimizing model structures to handle modality missingness, but models still face challenges in robustness when dealing with uncertain missingness.To this end, we propose a data-centric robust multimodal sentiment analysis method, Proxy-Driven Robust Multimodal Fusion (P-RMF).First, we map unimodal data to the latent space of Gaussian distributions to capture core features and structure, thereby learn stable modality representation.Then, we combine the quantified modality intrinsic uncertainty to learn stable multimodal joint representation (i.e., proxy modality), which is further enhanced through multi-layer dynamic cross-modal injection to increase its diversity.Extensive experimental results show that P-RMF outperforms existing models in noise resistance and achieves state-of-the-art performance on multiple benchmark datasets.

Multimodal Sentiment Analysis (MSA) is an attractive research that aims to integrate sentiment expressed in textual, visual, and acoustic signals. There are two main problems in the existing methods: 1) the dominant role of the text is underutilization in unaligned multimodal data, and 2) the modality under uncertain missing feature is not sufficiently explored. This paper proposes a Text-guided Reconstruction Network (TgRN) for MSA with uncertain missing modalities in non-aligned sequences. The TgRN network includes three primary modules: Text-guided Extraction Module (TEM), Reconstruction Module (RM) and Text-guided Fusion Module (TFM). First, the TEM consists of the text-guided cross attention units and self-attention units to capture inter-modal features and intra-modal features, respectively. Second, leveraging enhanced attention units and a three-way squeeze-and-excitation block, the RM is designed to learn semantic information from incomplete data and reconstruct missing modality features. Third, the TFM utilizes a progressive modality-mixing adaptation gate to explore the dynamic correlations between nonverbal and verbal modalities, effectively addressing the modality gap issue. Finally, under the supervision of sentiment prediction loss and reconstruction loss, the TgRN effectively processes both uncertain missing-modality conditions and ideal complete modality conditions. Extensive experiments on CMU-MOSI and CH-SIMS demonstrate that our proposed method outperforms state-of-the-art approaches.

Aspect-based Sentiment Analysis (ABSA) is a challenging and attractive fine-grained subtask in the Natural Language Processing (NLP) community, which aims to predict the sentiment polarity of each involved aspect term in the given sentence. With the numerous efforts contributed by related researchers, impressive success has been achieved in employing the external knowledge to enhance the sentiment representation. However, in previous works, the utilization of external knowledge is quite straightforward, which doesn't match the sentiment characteristics of human beings. Thus, this work proposes a novel humanized ABSA method, namely Fuzzy Humanized Sentiment-assisted Graph Attention Networks (FHSGAT). Preliminary, to acquire the essential semantic features, the sentence is re-constructed by the manually designed prompt template, which would assist the Pre-trained Language Model (PLM) in generating the target-guided semantic representation. Additionally, based on the normal dependency graph, triple-dimensional external knowledge is injected into the corresponding nodes. Specifically, this paper leverages the Gaussian membership function to fuzzify the sentiment knowledge-enhanced nodes, enabling the networks to obtain the humanized sentiment relationship between graph nodes, namely fuzzy humanized syntactic features. Besides, due to its exquisite performance in information fusion, Graph Attention Networks (GAT) is utilized to accomplish the integration of target-guided semantic representation and fuzzy humanized syntactic features synchronously. Eventually, to validate the effectiveness of the proposed method FHSGAT, extensive experiments are conducted on five publicly and widely used benchmarks, and the corresponding results show the performance of the proposed approach outperforms the state-of-the-art (SOTA) obviously.

Artificial intelligence methods offer objectivity and convenience in automatic depression detection, however, current research often neglects the critical role of facial landmarks. This oversight results in insufficient spatial structure information and a lack of detailed local representation, which fails to capture the nuanced semantic information crucial for identifying depression-related clues. To address these issues, we introduce a novel dual-branch network model comprising the Landmark-Image-Landmark Net (LIL Net) and the Global Context Vision Transformer Net (GCVit Net). Through a dual-stream, multiscale, and cross-fusion strategy, LIL Net is designed to extract original facial image features alongside landmark features, prioritizing the detailed semantic information of potential depression clues. LIL Net employs an innovative LIL Attention approach to jointly learn multiscale features from facial landmarks and images, thereby enhancing the model’s ability to capture fine-grained depression-related cues. Furthermore, the Multi-scale Feature Fusion (MSFF) module fuses the obtained multiscale features, augmenting the semantic expression of potential depression clues within facial landmarks via attention mechanisms. Meanwhile, the GCVit Net branch network supplements global information by extracting global facial features. Finally, the features from both branches are concatenated to enhance the accuracy of depression degree predictions. Experimental results demonstrate that our model has superior performance in detecting depression compared to existing methods. We release our code at https://github.com/xlx777/LIL-Net.

The goal of multilingual modelling is to generate multilingual text representations for various downstream tasks in different languages. However, some state-of-the-art pre-trained multilingual models perform poorly on many low-resource languages due to the lack of representation space and model capacity. To alleviate this issue, we propose a Multilingual model Enhanced with Visual Text Representations (MEVTR), which complements textual representations and extends the multilingual representation space with visual text representations. First, the visual encoder focuses on the glyphs and structure of the text to obtain visual text representations, and the textual encoder obtains textual representations. Then, multilingual representations are enhanced by aligning and fusing visual text representations and textual representations. Moreover, we propose similarity constraint, a self-supervised task to prompt the visual encoder to focus on more additional information. Prefix alignment and multi-head bilinear module are designed to acquire an improved integration effect of visual text representations and textual representations. Experimental results indicate that MEVTR benefits from visual text representations and achieves significant performance gains in downstream tasks. In particular, in the zero-shot cross-lingual transfer task, MEVTR achieves results that outperform the state-of-the-art adapter-based framework without the target language adapter.

In linguistics, all languages can be considered as symbolic systems, with each language relying on symbolic processes to associate specific symbols with meanings.In the same language, there is a fixed correspondence between linguistic symbol and meaning.In different languages, universal meanings follow varying rules of symbolization in one-to-one correspondence with symbols.Most work overlooks the properties of languages as symbol systems.In this paper, we shift the focus to the symbolic properties and introduce MTLS: a pre-training method to improve the multilingual capability of models by Making Texts into Linguistic Symbols.Initially, we replace the vocabulary in pre-trained language models by mapping relations between linguistic symbols and semantics.Subsequently, universal semantics within the symbolic system serve as bridges, linking symbols from different languages to the embedding space of the model, thereby enabling the model to process linguistic symbols.To evaluate the effectiveness of MTLS, we conducted experiments on multilingual tasks using BERT and RoBERTa, respectively, as the backbone.The results indicate that despite having just over 12,000 pieces of English data in pre-training, the improvement that MTLS brings to multilingual capabilities is remarkably significant.

Multimodal sentiment analysis (MSA) aims to integrate multiple modalities of information to better understand human sentiment. The current research mainly focuses on conducting multimodal fusion, which neglects the under-optimized modal representations generated by the imbalance of unimodal performances in joint learning. Moreover, the size of labeled datasets limits the generalization ability of existing supervised models. To address the above issues, this paper proposes a knowledge-enhanced self-supervised balanced representation approach (KEBR). First, a text-based cross-modal fusion method (TCMF) is constructed, which injects the non-verbal information from the videos into the semantic representation of text to enhance the multimodal representation of text. Then, a multimodal cosine constrained loss (MCC) is designed to constrain the fusion of non-verbal information in joint learning to balance the representation. Finally, with the help of sentiment knowledge and non-verbal information, KEBR conducts sentiment word masking and sentiment intensity prediction. Experimental results show that KEBR outperforms the baseline.

Summary In previous studies on facial video depression recognition, although convolutional neural network (CNN) has become a mainstream method, its performance still has room for improvement due to the insufficient extraction of global and local information and the neglect of the correlation of temporal and spatial information. This paper proposes a novel dual‐task enhanced global–local temporal–spatial network (DTE‐GLTS) to enhance the extraction capability of global and local features and deepen the analysis of temporal–spatial information correlation. We design a dual‐task learning mode that utilizes the data‐efficient image transformer (Deit) as the main body to learn the global features of video sequences and guides Deit to learn local features with the pre‐trained temporal–spatial fusion network (TSF). In addition, we propose the TSF mechanism to more effectively fuse temporal–spatial information in video sequences, strengthen the correlation between frames and pixels, and embed it in Resnet to form the TSF network. To the best of our knowledge, this is the first application of Deit and dual‐task learning mode in the field of facial video depression recognition. The experimental results on AVEC 2013 and AVEC 2014 show that our method achieves competitive performance, with mean absolute error/root mean square error (MAE/RMSE) scores of 6.06/7.73 and 5.91/7.68, respectively, while significantly reducing the number of parameters.

Depression is a prevalent and severe mental illness that significantly impacts patients’ physical health and daily life. Recent studies have focused on multimodal depression assessment, aiming to objectively and conveniently evaluate depression using multimodal data. However, existing methods based on audio–visual modalities struggle to capture the dynamic variations in depression clues and cannot fully explore multimodal data over a long time. In addition, they rely heavily on insufficient single-stage multimodal fusion, which limits the accuracy of depression assessment. To address these limitations, we propose a novel parallel multiscale bridge fusion network (PMBFN) for audio–visual depression assessment. PMBFN comprehensively captures subtle multilevel dynamic changes in depression expression through parallel multiscale dynamic convolutions and long short-term memories (LSTMs) and effectively solves the problem of long-term audio–visual sequence information loss by using spatiotemporal attention pooling modules. Furthermore, the multimodal bridge fusion module is proposed in PMBFN to achieve multistage interactive recursive multimodal fusion, enhancing the expressive capacity of multimodal depression-related features to improve the accuracy of assessment. Extensive experiments on the DAIC-WOZ and E-DAIC datasets demonstrate that our method outperforms current state-of-the-art methods and clearly shows our method's effectiveness eventually.

Behavior detection receives considerable attention in real-life human–computer interaction, where the complexity of background information and the variable durations of movements are two major factors affecting the accuracy of behavior detection. To overcome the inadequacy of these factors, this article proposes an enhancing region convolutional 3-D (ER-C3D) network with adaptive shrinkage and symmetrical multiscale for behavior detection. The improved ER-C3D network includes a feature subnet, a proposal subnet, and a classification subnet. First, a 3D-RSST unit is constructed by embedding an adaptive shrinkage structure and a soft thresholding operation. Meanwhile, a residual adaptive shrinkage mechanism, composed of multiple cascaded 3D-RSST units with different parameters, is designed to reduce redundant information of video streams in the feature subnet. Second, a spatiotemporal symmetrical multiscale structure is substituted for the single-layer convolution and embedded into the proposal subnet. Specially, contextual symmetrical multiscale motion characteristics with different levels and granularities are acquired by expanding the spatiotemporal receptive field of candidate temporal proposals. Finally, a soft-nonmaximal suppression strategy is introduced to filter high-quality temporal proposals in the classification subnet. The experimental results on the THUMOS’14 and ActivityNet1.2 datasets indicate that the [email protected] of the improved ER-C3D network reaches 39.4% and 42.2%, respectively, which is 10.5% and 15.4% higher than R-C3D. Compared with related methods, the proposed method shows improvement in both the positional precision of behavioral boundary and the accuracy of behavioral classification.

Human Multimodal Sentiment Analysis (MSA) is an attractive research that studies sentiment expressed from multiple heterogeneous modalities. While transformer-based methods have achieved great success, designing an effective “co-attention” model to associate text modality with nonverbal modalities remains challenging. There are two main problems: 1) the dominant role of the text in modalities is underutilization, and 2) the interaction between modalities is not sufficiently explored. This paper proposes a deep modular Co-Attention Shifting Network (CoASN) for MSA. A Cross-modal Modulation Module based on Co-attention (CMMC) and an Advanced Modality-mixing Adaptation Gate (AMAG) are constructed. The CMMC consists of the Text-guided Co-Attention (TCA) and Interior Transformer Encoder (ITE) units to capture inter-modal features and intra-modal features. With text modality as the core, the CMMC module aims to guide and promote the expression of emotion in nonverbal modalities, and the nonverbal modalities increase the richness of the text-based multimodal sentiment information. In addition, the AMAG module is introduced to explore the dynamical correlations among all modalities. Particularly, this efficient module first captures the nonverbal shifted representations and then combines them to calculate the shifted word embedding representations for the final MSA tasks. Extensive experiments on two commonly used datasets, CMU-MOSI and CMU-MOSEI, demonstrate that our proposed method is superior to the state-of-the-art performance.

Video-based dimensional emotion recognition, especially long sequence modeling, is a challenging task. This paper proposes a Multiscale Dynamic Temporal Convolution Network(MDTCN) for long sequence modeling of dimensional emotion recognition. Although Temporal Convolution Network(TCN) has performed well in sequence modeling, there are still some limitations that the feature scale of each layer is single and the receptive field of the upper layer is too large to capture the short-term dependence. Therefore, Dynamic Dilated Block(DDB) is constructed to better capture the temporal dependence by using dilated convolutions of different kernel size. In order to obtain both long-term and short-term temporal features, Multiscale TCN Structure is proposed. Extensive experiments are conducted on the AFFWILD2 and SEWA Dataset and the competitive result is shown compared to state-of-the-art methods.

Continuous dimensional emotion recognition for facial video sequence is a crucial and challenging task in Affective Computing and Human-Computer Intelligent Interaction. The key of this task is to effectively extract and discriminate spatial-temporal features in a more fine-grained way. In this paper, a Two-Stage Spatiotemporal Attention Temporal Convolution Network (TS-SATCN) is designed for continuous dimensional emotion recognition of facial videos. The first stage generates an initial recognition result that is later fed into the second for correction. In each stage, the introduced spatiotemporal attention branch helps the network learn different attention levels and focuses on the informative spatial-temporal features adaptively. The network is trained by a proposed smooth loss function which can further improve the predictions' quality. Extensive experiments are performed on two datasets, RECOLA and AFEW-VA, which shows that the proposed method achieves significant improvement over state-of-the-art methods.

To improve the space-time similarity and motion smoothness of facial expression imitation (FEI), a real-time FEI method for a humanoid robot is proposed based on smooth-constraint reversed mechanical model (SRMM) by combining a sequence-to-sequence deep learning model and a motion-smoothing constraint. First, on the basis of facial data from a Kinect capture device, a facial feature vector is characterized based on 3 head postures, 17 facial animation units, and facial geometric deformation cascaded by Laplace coordinates. Second, a reversed mechanical model is constructed via a multilayer long short-term memory neural network to accomplish direct mapping from facial feature sequences to motor position sequences. Additionally, to overcome the motor chattering phenomenon during real-time FEI, a high-order polynomial is constructed to fit the position sequence of motors, and an SRMM is proposed and designed based on the deviation of position, velocity, and acceleration. Finally, aiming to imitate the real-time facial feature sequences of a performer captured from Kinect, the optimal position sequences generated based on the SRMM is sent to the hardware system to keep the space-time characteristics consistent with those of the performer. The experimental results demonstrate that the motor position deviation of the SRMM is less than 8%. The space-time similarity between the robot and the performer is greater than 85%, and the motion smoothness of the online FEI exceeded 90%. Compared with other related methods, the proposed method achieves a remarkable improvement in motor position deviation, space-time similarity, and motion smoothness.