Tianhong Gao, Jundong Shen, Jiapeng Wang, Bei Shi, Ying Ju, Junfeng Yao, Huiyu Yu. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Multi-modal Multi-label Emotion Recognition (MMER) aims to identify various human emotions from heterogeneous visual, audio and text modalities. Previous methods mainly focus on projecting multiple modalities into a common latent space and learning an identical representation for all labels, which neglects the diversity of each modality and fails to capture richer semantic information for each label from different perspectives. Besides, associated relationships of modalities and labels have not been fully exploited. In this paper, we propose versaTile multi-modAl learning for multI-labeL emOtion Recognition (TAILOR), aiming to refine multi-modal representations and enhance discriminative capacity of each label. Specifically, we design an adversarial multi-modal refinement module to sufficiently explore the commonality among different modalities and strengthen the diversity of each modality. To further exploit label-modal dependence, we devise a BERT-like cross-modal encoder to gradually fuse private and common modality representations in a granularity descent way, as well as a label-guided decoder to adaptively generate a tailored representation for each label with the guidance of label semantics. In addition, we conduct experiments on the benchmark MMER dataset CMU-MOSEI in both aligned and unaligned settings, which demonstrate the superiority of TAILOR over the state-of-the-arts.

In Multi-View Multi-Label (MVML) learning, each instance is represented with several feature vectors and associated with multiple labels simultaneously. Existing models are mainly built on a shared latent space among multiple views and tend to select features that are distinguishable for all labels. However, they ignore the distinctive characteristic of each view and fail to extract the discriminative information related to corresponding labels. To address the above challenges, we proposed a novel approach named Label-specific Alignment with Adversarial Multi-view Representation (LAAMR). For one thing, shared and private information is extracted with adversarial training and orthogonal constraint, which ensures complementarity and consistency among multiple views. For another, label semantic information is well captured with multi-head attention, and then utilized as the guidance to exploit label-view dependence with alignment attention, resulting in label-specific representations. Experiments on various MVML datasets validate the effectiveness of LAAMR.

With the acceleration of urbanization, there is an explosive growth trend of the stroke burden in China. Consequently, it is necessary to adopt a regular screen to reduce brain damage and other complications. For one thing, the risk factors screened can be divided into 6 subsets: medical history, family history, fasting blood, lifestyle, physical characteristics, and demographic information. For another, the screen results can be categorized into 5 groups: normal carotid plaque, thickening, unstable plaque, and arterial stenosis. To address the above problems, Multi-View Multi-Label (MVML) learning provides a fundamental framework for handling such tasks, which treats the risk factors as multi-view data and the screen results as multi-label data. Besides, it is a challenging issue of how to effectively utilize view correlations and label correlations in MVML learning. In this paper, we propose a novel Multi-view Multi-label Learning with Dual-Attention Networks (MML-DAN) for stroke screen, which can strike a balance between label-specific views and label correlations. Meanwhile, the attention mechanism aims to focus on the most pertinent information of the corresponding labels rather than using all available risk factors equally. Experiments on one real-world dataset of stroke screen demonstrate the superiority of the proposed MML-DAN for stroke screen prediction. To further validate the effectiveness and generalization capability of MML-DAN, extensive experiments on 5 benchmark MVML datasets show that MML-DAN significantly outperforms other state-of-the-art MVML algorithms.

Multi-modal multi-label learning provides a fundamental framework for complex objects, which can be represented with multiple modalities and annotated with multiple labels simultaneously. Different modalities can usually provide complementary information, which may lead to improved performance. What's more, exploiting label correlations is crucially important to multi-label learning. However, most existing multi-label learning approaches do not sufficiently consider the complementary information among different modalities. In this paper, we propose a novel end-to-end deep learning framework named Rethinking Modal-oriented Label Correlations (RMLC), which sequentially polish the label prediction with each individual modality. In order to explicitly account for the correlated prediction of multiple labels, RMLC leverages an efficient sequential modal-based exploration to rethink label correlations. The final prediction of each label involves the collaboration between modal-specific prediction and the prediction of other labels based on cross-modal interaction. Comprehensive experiments on benchmark datasets validate the effectiveness and competitiveness of the proposed RMLC approach.