Multi-task learning (MTL) enables joint learning over multiple tasks based on shared representations, but suffers from task interference issue during optimization.Existing works mainly focus on task balancing or probabilistic modeling but fail to address the issue since they struggle to learn sufficient representations for all target tasks.To address this, we propose a multi-task representation alignment (MTRA) framework to achieve task-specific alignment and self-alignment on the shared representations from a mutual information perspective.MTRA ensures that the learned representations contain task-relevant features while mitigating the negative effects of task-irrelevant features.First, we design a task-specific alignment objective to align the shared representations and task-specific representations with the expected targets of all tasks via information maximization.Besides, we design a self-alignment objective to eliminate task-irrelevant features via conditional information minimization.Experiments on two multi-task language benchmarks show that MTRA outperforms 13 representative MTL methods under the same settings, particularly under label-noisy and dataconstrained conditions.Further analysis shows that the learned shared representations exhibit sufficient task informativeness and superior alignment properties.

This paper proposes a new principled multi-task representation learning framework (InfoMTL) to extract noise-invariant sufficient representations for all tasks. It ensures sufficiency of shared representations for all tasks and mitigates the negative effect of redundant features, which can enhance language understanding of pre-trained language models (PLMs) under the multi-task paradigm. Firstly, a shared information maximization principle is proposed to learn more sufficient shared representations for all target tasks. It can avoid the insufficiency issue arising from representation compression in the multi-task paradigm. Secondly, a task-specific information minimization principle is designed to mitigate the negative effect of potential redundant features in the input for each task. It can compress task-irrelevant redundant information and preserve necessary information relevant to the target for multi-task prediction. Experiments on six classification benchmarks show that our method outperforms 12 comparative multi-task methods under the same multi-task settings, especially in data-constrained and noisy scenarios. Extensive experiments demonstrate that the learned representations are more sufficient, data-efficient, and robust.

Multi-task learning (MTL) enhances efficiency by sharing representations across tasks, but task dissimilarities often cause partial learning, where some tasks dominate while others are neglected.Existing methods mainly focus on balancing loss or gradients but fail to fundamentally address this issue due to the representation discrepancy in latent space.In this paper, we propose variance-invariant probabilistic decoding for multi-task learning (VIP-MTL), a framework that ensures impartial learning by harmonizing representation spaces across tasks.VIP-MTL decodes shared representations into task-specific probabilistic distributions and applies variance normalization to constrain these distributions to a consistent scale.Experiments on two language benchmarks show that VIP-MTL outperforms 12 representative methods under the same multi-task settings, especially in heterogeneous task combinations and dataconstrained scenarios.Further analysis shows that VIP-MTL is robust to sampling distributions, efficient on optimization process, and scale-invariant to task losses.Additionally, the learned task-specific representations are more informative, enhancing the language understanding abilities of pre-trained language models under the multi-task paradigm.

This paper presents a new supervised representation learning framework, namely structured probabilistic coding (SPC), to learn compact and informative representations from input related to the target task. SPC is an encoder-only probabilistic coding technology with a structured regularization from the target space. It can enhance the generalization ability of pre-trained language models for better language understanding. Specifically, our probabilistic coding simultaneously performs information encoding and task prediction in one module to more fully utilize the effective information from input data. It uses variational inference in the output space to reduce randomness and uncertainty. Besides, to better control the learning process of probabilistic representations, a structured regularization is proposed to promote uniformity across classes in the latent space. With the regularization term, SPC can preserve the Gaussian structure of the latent code and achieve better coverage of the hidden space with class uniformly. Experimental results on 12 natural language understanding tasks demonstrate that our SPC effectively improves the performance of pre-trained language models for classification and regression. Extensive experiments show that SPC can enhance the generalization capability, robustness to label noise, and clustering quality of output representations.

This paper proposes an information-theoretic representation learning framework, named conditional information flow maximization, to extract noise-invariant sufficient representations for the input data and target task.It promotes the learned representations have good feature uniformity and sufficient predictive ability, which can enhance the generalization of pre-trained language models (PLMs) for the target task.Firstly, an information flow maximization principle is proposed to learn more sufficient representations for the input and target by simultaneously maximizing both inputrepresentation and representation-label mutual information.Unlike the information bottleneck, we handle the input-representation information in an opposite way to avoid the overcompression issue of latent representations.Besides, to mitigate the negative effect of potential redundant features from the input, we design a conditional information minimization principle to eliminate negative redundant features while preserve noise-invariant features.Experiments on 13 language understanding benchmarks demonstrate that our method effectively improves the performance of PLMs for classification and regression.Extensive experiments show that the learned representations are more sufficient, robust and transferable.

Emotional support conversation (ESC) task aims to relieve the emotional distress of users who have high-intensity of negative emotions. However, due to the ignorance of emotion intensity modelling which is essential for ESC, previous methods fail to capture the transition of emotion intensity effectively. To this end, we propose a Multi-stream information Fusion Framework (MFF-ESC) to thoroughly fuse three streams (text semantics stream, emotion intensity stream, and feedback stream) for the modelling of emotion intensity, based on a designed multi-stream fusion unit. As the difficulty of modelling subtle transitions of emotion intensity and the strong emotion intensity-feedback correlations, we use the KL divergence between feedback distribution and emotion intensity distribution to further guide the learning of emotion intensities. Experimental results on automatic and human evaluations indicate the effectiveness of our method.

Many fake news detection studies have achieved promising performance by extracting effective semantic and structure features from both content and propagation trees. However, it is challenging to apply them to practical situations, especially when using the trained propagation-based models to detect news with no propagation data. Towards this scenario, we study a new task named cold-start fake news detection, which aims to detect content-only samples with missing propagation. To achieve the task, we design a simple but effective Structure Adversarial Net (SAN) framework to learn transferable features from available propagation to boost the detection of content-only samples. SAN introduces a structure discriminator to estimate dissimilarities among learned features with and without propagation, and further learns structure-invariant features to enhance the generalization of existing propagation-based methods for content-only samples. We conduct qualitative and quantitative experiments on three datasets. Results show the challenge of the new task and the effectiveness of our SAN framework.

Automatic rumor detection is critical for maintaining a healthy social media environment. The mainstream methods generally learn rich features from information cascades by modeling the cascade as a tree or graph structure where edges are built based on interactions between a tweet and retweets. Some psychology studies have empirically shown that users' various subjective factors always cause the uncertainty of interactions such as differences among interactive behavior activation thresholds or semantic relevancy. However, previous works model interactions by employing a simple fully connected layer on fixed edge weights in the graph and cannot reasonably describe this inherent uncertainty of complex interactions. In this article, inspired by the fuzzy theory, we propose a novel neuro-fuzzy method, fuzzy graph convolutional networks (FGCNs), to sufficiently understand uncertain interactions in the information cascade in a fuzzy perspective. Specifically, a new strategy of graph construction is first designed to convert each information cascade into a heterogeneous graph structure with the consideration of explicit interactive behaviors between a tweet and its retweet, as well as implicit interactive behaviors among retweets, enriching more structural clues in the graph. Then, we improve graph convolutional networks by incorporating edge fuzzification (EF) modules. The EFs adapt edge weights according to predefined membership to enhance message passing in the graph. The proposed model can provide a stronger relational inductive bias for expressing uncertain interactions and capture more discriminative and robust structural features for rumor detection. Extensive experiments demonstrate the effectiveness and superiority of FGCN on both rumor detection and early rumor detection.

Extracting generalized and robust representations is a major challenge in emotion recognition in conversations (ERC).To address this, we propose a supervised adversarial contrastive learning (SACL) framework for learning classspread structured representations in a supervised manner.SACL applies contrast-aware adversarial training to generate worst-case samples and uses joint class-spread contrastive learning to extract structured representations.It can effectively utilize label-level feature consistency and retain fine-grained intra-class features.To avoid the negative impact of adversarial perturbations on context-dependent data, we design a contextual adversarial training (CAT) strategy to learn more diverse features from context and enhance the model's context robustness.Under the framework with CAT, we develop a sequence-based SACL-LSTM to learn label-consistent and context-robust features for ERC.Experiments on three datasets show that SACL-LSTM achieves state-of-the-art performance on ERC.Extended experiments prove the effectiveness of SACL and CAT.

Multimodal fake news detection has obtained increasing attention recently. Existing works generally encode multimodal contents into a deterministic point in semantic subspaces, and then fuse multimodal features by simple concatenation or attention mechanisms. However, most methods suffer from adapting to noisy multimodal contents since they neglect the robustness of modality-specific features. Besides, as different modalities usually have varying confidence levels, previous attention-based fusion models that learn modality-independent weights based on the input data feature, would limit the optimal integration of multimodal contents. To alleviate the above issues, we propose novel Multimodal Uncertainty Learning Network (MM-ULN) to enhance multimodal fake news detection by modeling both intra- and inter-modality uncertainty. Specifically, we incorporate a novel intra-modality uncertainty learning (EUL) module to better understand noisy multimodal contents. EULs provide feature regularization in a variational way, successfully alleviating the effects of data uncertainty within modalities. We design a new variational attention fusion (VAF) module to adaptively fuse multimodal contents with modality-dependent weights. The VAF module consider the relative confidence between modalities and enables to explore complementary properties for detection. Extensive experiments on two benchmark datasets demonstrate the effectiveness and superiority of MM-ULN on multimodal fake news detection.

The widespread of fake news has detrimental societal effects. Recent works model information propagation as graph structure and aggregate structural features from user interactions for fake news detection. However, they usually neglect a broader propagation uncertainty issue, caused by some missing and unreliable interactions during actual spreading, and suffer from learning accurate and diverse structural properties. In this paper, we propose a novel dual graph-based model, Uncertainty-aware Propagation Structure Reconstruction (UPSR) for improving fake news detection. Specifically, after the original propagation modeling, we introduce propagation structure reconstruction to fully explore latent interactions in the actual propagation. We design a novel Gaussian Propagation Estimation to refine the original deterministic node representation by multiple Gaussian distributions and arise latent interactions with KL divergence between distributions in a multi-facet manner. Extensive experiments on two real-world datasets demonstrate the effectiveness and superiority of our model.

Fake news’s quick propagation on social media brings severe social ramifications and economic damage. Previous fake news detection usually learn semantic and structural patterns within a single target propagation tree. However, they are usually limited in narrow signals since they do not consider latent information cross other propagation trees. Motivated by a common phenomenon that most fake news is published around a specific hot event/topic, this paper develops a new concept of propagation forest to naturally combine propagation trees in a semantic-aware clustering. We propose a novel Unified Propagation Forest-based framework (UniPF) to fully explore latent correlations between propagation trees to improve fake news detection. Besides, we design a root-induced training strategy, which encourages representations of propagation trees to be closer to their prototypical root nodes. Extensive experiments on four benchmarks consistently suggest the effectiveness and scalability of UniPF.

Emotion Recognition in Conversations (ERC) has considerable prospects for developing empathetic machines. For multimodal ERC, it is vital to understand context and fuse modality information in conversations. Recent graph-based fusion methods generally aggregate multimodal information by exploring unimodal and cross-modal interactions in a graph. However, they accumulate redundant information at each layer, limiting the context understanding between modalities. In this paper, we propose a novel Multimodal Dynamic Fusion Network (MM-DFN) to recognize emotions by fully understanding multimodal conversational context. Specifically, we design a new graph-based dynamic fusion module to fuse multimodal context features in a conversation. The module reduces redundancy and enhances complementarity between modalities by capturing the dynamics of contextual information in different semantic spaces. Extensive experiments on two public benchmark datasets demonstrate the effectiveness and superiority of the proposed model.

Dou Hu, Lingwei Wei, Xiaoyong Huai. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Lingwei Wei, Dou Hu, Wei Zhou, Zhaojuan Yue, Songlin Hu. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.