Recent studies have explored Continual Instruction Tuning (CIT) in Multimodal Large Language Models (MLLMs), with a primary focus on Task-incremental CIT, where MLLMs are required to continuously acquire new tasks.However, the more practical and challenging Domain-incremental CIT, focused on the continual adaptation of MLLMs to new domains, remains underexplored.In this paper, we propose a new Sparse Mixture of Expert (SMoE) based method for domain-incremental CIT in MLLMs.During training, we learn a domainspecific SMoE module for each new domain in every FFN sub-layer of MLLMs, preventing catastrophic forgetting caused by inter-domain conflicts.Moreover, we equip the SMoE module with a domain-specific autoregressive loss (DSAL), which is used to identify the most suitable SMoE module for processing each test instruction during inference.To further enhance the SMoE module's ability to learn domain knowledge, we design an adaptive thresholdbased router (AT-Router) that allocates computing resources (experts) to instruction tokens based on their importance.Finally, we establish a new benchmark to evaluate the efficacy of our method and advance future research.Extensive experiments show that our method consistently outperforms all competitive baselines.

The few-shot relation extraction (FSRE) aims at enhancing the model's generalization to new relations with a few labeled instances.Existing studies usually adopt prototype networks (Pro-toNets) for FSRE and assume that the support set, adapting the model to new relations, only contains accurately labeled support instances.However, this assumption is often unrealistic, as even carefully-annotated datasets commonly contain mislabeled instances.In this paper, we first conduct a preliminary study that reveals the high sensitivity of ProtoNets to noisy labels in the support set.Moreover, we find that fully leveraging mislabeled support instances is crucial for enhancing the FSRE model robustness.Thus, we propose a self-denoising model for FSRE, designed to improve model robustness by automatically correcting mislabeled support instances.It comprises two core components: 1) a label correction module (LCM), used to correct noisy labels of support instances based on the distances between them in the embedding space, and 2) a relation classification module (RCM), aimed to achieve more accurate predictions for new relations based on the corrected labels produced by LCM.Moreover, we propose a feedback-based training strategy, which focuses on training LCM and RCM to synergistically handle noisy labels in support set.Experimental results on two public datasets confirm the robustness of our model.Particularly, even in scenarios without noisy labels, our model significantly outperforms all baselines.

Liang Zhang, Zhen Yang, Biao Fu, Ziyao Lu, Liangying Shao, Shiyu Liu, Fandong Meng, Jie Zhou, Xiaoli Wang, Jinsong Su. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

k-Nearest-Neighbor Machine Translation (kNN-MT) becomes an important research direction of NMT in recent years.Its main idea is to retrieve useful key-value pairs from an additional datastore to modify translations without updating the NMT model.However, the underlying retrieved noisy pairs will dramatically deteriorate the model performance.In this paper, we conduct a preliminary study and find that this problem results from not fully exploiting the prediction of the NMT model.To alleviate the impact of noise, we propose a confidence-enhanced kNN-MT model with robust training.Concretely, we introduce the NMT confidence to refine the modeling of two important components of kNN-MT: kNN distribution and the interpolation weight.Meanwhile we inject two types of perturbations into the retrieved pairs for robust training.Experimental results on four benchmark datasets demonstrate that our model not only achieves significant improvements over current kNN-MT models, but also exhibits better robustness.Our code is available at https://github.com/ DeepLearnXMU/Robust-knn-mt.

We present CLIDSUM, a benchmark dataset towards building cross-lingual summarization systems on dialogue documents.It consists of 67k+ dialogue documents and 112k+ annotated summaries in different target languages.Based on the proposed CLIDSUM, we introduce two benchmark settings for supervised and semi-supervised scenarios, respectively.We then build various baseline systems in different paradigms (pipeline and end-to-end) and conduct extensive experiments on CLIDSUM to provide deeper analyses.Furthermore, we propose mDIALBART which extends mBART via further pre-training, where the multiple objectives help the pre-trained model capture the structural characteristics as well as key content in dialogues and the transformation from source to the target language.Experimental results show the superiority of mDIALBART, as an end-to-end model, outperforms strong pipeline models on CLIDSUM.Finally, we discuss specific challenges that current approaches faced with this task and give multiple promising directions for future research.

Existing studies for multi-source neural machine translation (NMT) either separately model different source sentences or resort to the conventional single-source NMT by simply concatenating all source sentences. However, there exist two drawbacks in these approaches. First, they ignore the explicit word-level semantic interactions between source sentences, which have been shown effective in the embeddings of multilingual texts. Second, multiple source sentences are simultaneously encoded by an NMT model, which is unable to fully exploit the semantic information of each source sentence. In this paper, we explore multi-stage information interactions for multi-source NMT. Specifically, we first propose a multi-source NMT model that performs information interactions at the encoding stage. Its encoder contains multiple semantic interaction layers, each of which sequentially consists of (1) monolingual semantic interaction sub-layer, which is based on the self-attention mechanism and used to learn word-level monolingual contextual representations of source sentences, and (2) cross-lingual semantic interaction sub-layer, which leverages word alignments to perform fine-grained semantic transitions among hidden states of different source sentences. Furthermore, at the training stage, we introduce a mutual distillation based training framework, where single-source models and ours perform information interactions. Such framework can fully exploit the semantic information of each source sentence to enhance our model. Extensive experimental results on the WMT14 English-German-French dataset show our method exhibits significant improvements upon competitive baselines.

Yubin Ge, Ly Dinh, Xiaofeng Liu, Jinsong Su, Ziyao Lu, Ante Wang, Jana Diesner. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

In aspect-level sentiment classification (ASC), it is prevalent to equip dominant neural models with attention mechanisms, for the sake of acquiring the importance of each context word on the given aspect. However, such a mechanism tends to excessively focus on a few frequent words with sentiment polarities, while ignoring infrequent ones. In this paper, we propose a progressive self-supervised attention learning approach for neural ASC models, which automatically mines useful attention supervision information from a training corpus to refine attention mechanisms. Specifically, we iteratively conduct sentiment predictions on all training instances. Particularly, at each iteration, the context word with the maximum attention weight is extracted as the one with active/misleading influence on the correct/incorrect prediction of every instance, and then the word itself is masked for subsequent iterations. Finally, we augment the conventional training objective with a regularization term, which enables ASC models to continue equally focusing on the extracted active context words while decreasing weights of those misleading ones. Experimental results on multiple datasets show that our proposed approach yields better attention mechanisms, leading to substantial improvements over the two state-of-the-art neural ASC models. Source code and trained models are available at https://github.com/DeepLearnXMU/PSSAttention.