Sparse Mixture-of-Experts (SMoE) architectures have enabled a new frontier in scaling Large Language Models (LLMs), offering superior performance by activating only a fraction of their total parameters during inference. However, their practical deployment is severely hampered by substantial static memory overhead, as all experts must be loaded into memory. Existing post-training pruning methods, while reducing model size, often derive their pruning criteria from a single, general-purpose corpus. This leads to a critical limitation: a catastrophic performance degradation when the pruned model is applied to other domains, necessitating a costly re-pruning for each new domain. To address this generalization gap, we introduce Mosaic Pruning (MoP). The core idea of MoP is to construct a functionally comprehensive set of experts through a structured ``cluster-then-select" process. This process leverages a similarity metric that captures expert performance across different task domains to functionally cluster the experts, and subsequently selects the most representative expert from each cluster based on our proposed Activation Variability Score. Unlike methods that optimize for a single corpus, our proposed Mosaic Pruning ensures that the pruned model retains a functionally complementary set of experts, much like the tiles of a mosaic that together form a complete picture of the original model's capabilities, enabling it to handle diverse downstream tasks.Extensive experiments on various MoE models demonstrate the superiority of our approach. MoP significantly outperforms prior work, achieving a 7.24\% gain on general tasks and 8.92\% on specialized tasks like math reasoning and code generation.

Spoken Language Understanding (SLU) consists of two sub-tasks: intent detection (ID) and slot filling (SF). Given its broad range of real-world applications, enhancing SLU for practical deployment is increasingly critical. Profile-based SLU addresses ambiguous user utterances by incorporating context awareness (CA), user profiles (UP), and knowledge graphs (KG) to support disambiguation, thereby advancing SLU research toward real-world applicability. However, existing SLU datasets still fall short in representing real-world scenarios. Specifically, (1) CA uses one-hot vectors for representation, which is overly idealized, and (2) models typically focuses solely on predicting intents and slot labels, neglecting the reasoning process that could enhance performance and interpretability. To overcome these limitations, we introduce VRSLU, a novel SLU dataset that integrates both Visual images and explicit Reasoning. For over-idealized CA, we use GPT-4o and FLUX.1-dev to generate images reflecting users’ environments and statuses, followed by human verification to ensure quality. For reasoning, GPT-4o is employed to generate explanations for predicted labels, which are then refined by human annotators to ensure accuracy and coherence. Additionally, we propose an instructional template, LR-Instruct, which first predicts labels and then generates corresponding reasoning. This two-step approach helps mitigate the influence of reasoning bias on label prediction. Experimental results confirm the effectiveness of incorporating visual information and highlight the promise of explicit reasoning in advancing SLU.

Wentao Hu, Yanbo Zhai, Xiaohui Hu, Mingkuan Zhao, Shanhong yu, Xue Liu, Kaidong Yu, Shuangyong Song, Xuelong Li. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Jie Zhang, Changzai Pan, Sishi Xiong, Kaiwen Wei, Yu Zhao, Xiangyu Li, Jiaxin Peng, Xiaoyan Gu, Jian Yang, Wenhan Chang, Zhenhe Wu, Jiang Zhong, Shuangyong Song, Xuelong Li. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Empathy detection (ED) and empathy intent classification (EIC) aim to identify the empathy direction expressed in user utterances and the underlying empathy intent behind them. Previous studies show that facilitating information transfer between tasks can enhance model performance. However, the interaction between ED and EIC in few-shot learning remains underexplored. To this end, we identify the challenges in jointly training ED and EIC in a few-shot setting: establishing effective information transfer between them and improving the model’s generalization capability. We propose a novel model called USB. For information transfer, the interactive module maps empathy and empathy intent labels through utterances to model task correlations. For generalization capability, after capturing empathy and empathy intent representations with an adaptive fusion module, we introduce a multi-level contrastive learning strategy to optimize representations at task and label levels, enhancing generalization. Experimental results on two public datasets show that our model outperforms all baselines.

Empathy detection (ED) aims to understand the user’s empathy direction, while empathy intent classification (EIC) focuses on identifying the empathy intent behind the user’s utterance. Both tasks have garnered significant attention. Recent studies have shown that jointly training these tasks can improve model performance, as their correlation enhances the diversity of information. However, previous studies have relied solely on shallow information transfer between two task representations, failing to fully leverage the inter-task correlation, thus limiting performance. To this end, we propose a novel Label Co-occurrence Transformation Network (LCoT-Net), which models the correlation between the two tasks using the co-occurrence matrix of empathy and empathy intent labels as a medium. By performing category feature transformation at both the label and utterance levels, we achieve two-level mutual task guidance. Experimental results demonstrate that our model achieves competitive performance across various settings on two public datasets.

Domain generalization aims to learn invariance across multiple source domains, thereby enhancing generalization against out-of-distribution data. While gradient or representation matching algorithms have achieved remarkable success in domain generalization, these methods generally lack generalization guarantees or depend on strong assumptions, leaving a gap in understanding the underlying mechanism of distribution matching. In this work, we formulate domain generalization from a novel probabilistic perspective, ensuring robustness while avoiding overly conservative solutions. Through comprehensive information-theoretic analysis, we provide key insights into the roles of gradient and representation matching in promoting generalization. Our results reveal the complementary relationship between these two components, indicating that existing works focusing solely on either gradient or representation alignment are insufficient to solve the domain generalization problem. In light of these theoretical findings, we introduce IDM to simultaneously align the inter-domain gradients and representations. Integrated with the proposed PDM method for complex distribution matching, IDM achieves superior performance over various baseline methods.

Pre-trained language models have shown their high performance of text processing in intelligent customer service platforms. However, these models do not leverage domain specific information. In this paper, we propose icsPLMs optimized for intelligent customer service on both word and sentence levels. Our experimental results represent that using targeted strategies can further improve the performance of pre-trained language models in this field.

Large-scale pre-trained neural language model has facilitated to achieve the state-of-the-art performance on Dialogue State Tracking (DST) tasks. One of the existing works models the semantic correlation between the dialogue context and (domain, slot) pair encoded by BERT and make the prediction. Despite the effectiveness, they ignore the fact that there is no perfect semantic correspondence between (domain, slot) pair and the dialogue context. In this paper, we propose a domain-slot aware contrastive learning framework to solve this problem, which proposes three methods to bridge the semantic gap between the dialogue context and the (domain, slot) by constructing training sample pairs to fine-tune the BERT model and use it for base DST model. The experiments demonstrate that our proposed method has improved the performance of the baseline model on the MultiWOZ2.1 and MultiWOZ2.4 datasets, yielding competitive results.

The recent surge in contrastive learning has intensified the interest in understanding the generalization of non-pointwise learning paradigms. While information-theoretic analysis achieves remarkable success in characterizing the generalization behavior of learning algorithms, its applicability is largely confined to pointwise learning, with extensions to the simplest pairwise settings remaining unexplored due to the challenges of non-i.i.d losses and dimensionality explosion. In this paper, we develop the first series of information-theoretic bounds extending beyond pointwise scenarios, encompassing pointwise, pairwise, triplet, quadruplet, and higher-order scenarios, all within a unified framework. Specifically, our hypothesis-based bounds elucidate the generalization behavior of iterative and noisy learning algorithms via gradient covariance analysis, and our prediction-based bounds accurately estimate the generalization gap with computationally tractable low-dimensional information metrics. Comprehensive numerical studies then demonstrate the effectiveness of our bounds in capturing the generalization dynamics across diverse learning scenarios.

Dialogue state tracking is responsible for predicting the user’s dialogue state during the whole dialogue process. In practical applications, values for different slots exist in individual utterances of the dialog history. With the accumulation of the dialogue history, it becomes extremely difficult to accurately predict slots and corresponding values from the lengthy dialogue history. To solve the problem of the interference caused by lengthy dialogue history, we propose a dual hierarchical selective augmentation method, which makes use of two hierarchical level information selection strategy to generate slot values. In the encoding phase, we first extract word-level matching features between the slot and each dialogue turn, and then build turn-level context relevance. In the decoding phase, first of all, from a global perspective, the dialogue turn information is selected multiple according to the dialogue context and slot, so that the model focuses more on the turn containing slot value. Secondly, our model performs weighted context attention to capture the critical words of dialogue turn from the local view. This dual hierarchical context selection alleviates the interference caused by excessive redundant information in the dialogue history and enhances the judgment ability of the model for vital turns and words. Furthermore, to enhance the copying ability of the model, we use the turn selection-guided pointer network to copy slot values from the dialogue. Experimental results show that our model significantly outperforms multiple baselines on the released MultiWOZ benchmark.

The Dialogue State Tracking (DST) module tracks the user’s intent by populating multiple predefined slots related to the dialogue task. In recent years, various graph neural network-based DST methods have been proposed to establish graph structures capturing the correlations between domains and slots, thereby enhancing model performance. However, these methods may involve redundant connections in the graph structure. To better construct relationships between domains and slots, we introduce a graph neural network-based dialogue state tracking method called Dynamic Domain Selection Graph DST (DDSG-DST). Specifically, (1) we employ Graphormer to establish hierarchical relationships between domains and slots; (2) we propose an additional domain prediction auxiliary task to predict the domain relevant to the dialogue context; (3) based on the predicted relevant domain from the auxiliary task, we dynamically select domain node information in the graph and perform dialogue state prediction. Experimental results demonstrate that we effectively establish hierarchical relationships between domains and slots, mitigate the negative impact of redundant connections in the graph structure, and enhance model performance.

In our dynamic world where data arrives in a continuous stream, continual learning enables us to incrementally add new tasks/domains without the need to retrain from scratch. A major challenge in continual learning of language model is catastrophic forgetting, the tendency of models to forget knowledge from previously trained tasks/domains when training on new ones. This paper studies dialog generation under the continual learning setting. We propose a novel method that 1) uses Text-Mixup as data augmentation to avoid model overfitting on replay memory and 2) leverages Batch-Nuclear Norm Maximization (BNNM) to alleviate the problem of mode collapse. Experiments on a 37-domain task-oriented dialog dataset and DailyDialog (a 10-domain chitchat dataset) demonstrate that our proposed approach outperforms the state-of-the-art in continual learning.

Open-domain multi-modal dialogue system heavily relies on visual information to generate contextually relevant responses. The existing open-domain multi-modal dialog generation methods ignore the complementary relationship between multiple modalities, and are difficult to integrate with LLMs. To tackle these challenges, we introduce AutoGraph, an innovative method for constructing visual context graphs automatically. We aim to structure complex information and seamlessly integrate it with large language models (LLMs), aligning information from multiple modalities at both semantic and structural levels. Specifically, we fully connect the text graphs and scene graphs, and then trim unnecessary edges via LLMs to automatically construct a visual context graph. Next, we design several graph sampling grammar for the first time to convert graph structures into sequence which is suitable for LLMs. Finally, we propose a two-stage fine-tuning strategy to allow LLMs to understand graph sampling grammar and generate responses. We validate our proposed method on text-based LLMs, and visual-based LLMs, respectively. Experimental results show that our proposed method achieves state-of-the-art performance on multiple public datasets.

Although current Graph Neural Network (GNN) based models achieved good performances in Dialogue Intent Classification (DIC), they leaf the inherent domain-specific knowledge out of consideration, leading to the lack of ability of acquiring fine-grained semantic information. In this paper, we propose a Knowledge-Enhanced Multifactor Graph (KEMG) Model for DIC. We firstly present a knowledge-aware utterance encoder with the help of a domain-specific knowledge graph, fusing token-level and entity-level semantic information, then design a heterogeneous dialogue graph encoder by explicitly modeling several factors that matter to contextual modeling of dialogues. Experiment results show that our proposed method outperforms other GNN-based methods on a dataset collected from a real-world online customer service dialogue system on the e-commerce website, JD.

Dialogue state error correction has recently been proposed to correct wrong slot values in predicted dialogue states, thereby mitigating the error propagation problem for dialogue state tracking (DST).These approaches, though effective, are heavily intertwined with specific DST models, limiting their applicability to other DST models.To solve this problem, we propose Scalable Dialogue State Correction (Scalable-DSC), which can correct wrong slot values in the dialogue state predicted by any DST model.Specifically, we propose a Structural Template Prompt (STP) that converts predicted dialogue state from any DST models into a standardized natural language sequence as a part of the historical context, associates them with dialogue history information, and generates a corrected dialogue state sequence based on predefined template options.We further enhance Scalable-DSC by introducing two training strategies.The first employs a predictive state simulator to simulate the predicted dialogue states as the training data to enhance the generalization ability of the model.The second involves using the dialogue state predicted by DST as the training data, aiming at mitigating the inconsistent error type distribution between the training and inference.Experiments confirm that our model achieves state-of-the-art results on MultiWOZ 2.0-2.4 △ .

Session-based Recommendation (SR) aims to predict users' next click based on their behavior within a short period, which is crucial for online platforms. However, most existing SR methods somewhat ignore the fact that user preference is not necessarily strongly related to the order of interactions. Moreover, they ignore the differences in importance between different samples, which limits the model-fitting performance. To tackle these issues, we put forward the method, Mining Interest Trends and Adaptively Assigning Sample Weight, abbreviated as MTAW. Specifically, we model users' instant interest based on their present behavior and all their previous behaviors. Meanwhile, we discriminatively integrate instant interests to capture the changing trend of user interest to make more personalized recommendations. Furthermore, we devise a novel loss function that dynamically weights the samples according to their prediction difficulty in the current epoch. Extensive experimental results on two benchmark datasets demonstrate the effectiveness and superiority of our method.

Customer service is very important in many industrial fields, and the service quality is most essential. However, customer service practitioners are with a high turnover rate, and it usually takes months for a new customer service employee to be an experienced one. If the training of new employees is conducted by other experienced employees, there will be a high resource consumption. Therefore, intelligent training systems for customer service can be designed to replace the manual training. In this paper, we define the task of intelligent training for customer service and propose an architecture of intelligent training systems. Dialogue scripts are prepared offline, and a dialogue simulation module and a service evaluation module are separately intended for the online service training and the service quality evaluation. We evaluate state-of-the-art models with respect to the ability to provide service training, and the experimental results show that our proposed system is effective on this task.

Abstract Hierarchical Text Classification (HTC) is a formidable task which involves classifying textual descriptions into a taxonomic hierarchy. Existing methods, however, have difficulty in adequately modeling the hierarchical label structures, because they tend to focus on employing graph embedding methods to encode the hierarchical structure while disregarding the fact that the HTC labels are rooted in a tree structure. This is significant because, unlike a graph, the tree structure inherently has a directive that ordains information flow from one node to another—a critical factor when applying graph embedding to the HTC task. But in the graph structure, message-passing is undirected, which will lead to the imbalance of message transmission between nodes when applied to HTC. To this end, we propose a unidirectional message-passing multi-label generation model for HTC, referred to as UMP-MG. Instead of viewing HTC as a classification problem as previous methods have done, this novel approach conceptualizes it as a sequence generation task, introducing prior hierarchical information during the decoding process. This further enables the blocking of information flow in one direction to ensure that the graph embedding method is better suited for the HTC task and thus resulted in the enhanced tree structure representation. Results obtained through experimentation on both the public WOS dataset and an E-commerce user intent classification dataset demonstrate that our proposed model can achieve superlative results.

This paper presents SimCTC, a simple contrastive learning (CL) framework that greatly advances the state-of-the-art text clustering models. In SimCTC, a pre-trained BERT model first maps the input sequence to the representation space, which is then followed by three different loss function heads: Clustering head, Instance-CL head and Cluster-CL head. Experimental results on multiple benchmark datasets demonstrate that SimCTC remarkably outperforms 6 competitive text clustering methods with 1%-6% improvement on Accuracy (ACC) and 1%-4% improvement on Normalized Mutual Information (NMI). Moreover, our results also show that the clustering performance can be further improved by setting an appropriate number of clusters in the cluster-level objective.

Intent identification is the key technology in dialogue system. However, not all online queries are clear or complete. To identify users' intents from those fuzzy queries accurately, this paper proposes a multi-factor classification framework on the query level. Experimental results on our online serving system JIMI demonstrate the effectiveness of our proposed framework.

Due to the increasing use of service chatbots in E-commerce platforms in recent years, customer satisfaction prediction (CSP) is gaining more and more attention. CSP is dedicated to evaluating subjective customer satisfaction in conversational service and thus helps improve customer service experience. However, previous methods focus on modeling customer-chatbot interaction across different turns, which are hard to represent the important dynamic satisfaction states throughout the customer journey. In this work, we investigate the problem of satisfaction states tracking and its effects on CSP in E-commerce service chatbots. To this end, we propose a dialogue-level classification model named DialogueCSP to track satisfaction states for CSP. In particular, we explore a novel two-step interaction module to represent the dynamic satisfaction states at each turn. In order to capture dialogue-level satisfaction states for CSP, we further introduce dialogue-aware attentions to integrate historical informative cues into the interaction module. To evaluate the proposed approach, we also build a Chinese E-commerce dataset for CSP. Experiment results demonstrate that our model significantly outperforms multiple baselines, illustrating the benefits of satisfaction states tracking on CSP.

Hongyan Xie, Haoxiang Su, Shuangyong Song, Hao Huang, Bo Zou, Kun Deng, Jianghua Lin, Zhihui Zhang, Xiaodong He. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

This paper aims to predict user satisfaction for customer service chatbot in session level, which is of great practical significance yet rather untouched. It requires to explore the relationship between questions and answers across different rounds of interactions, and handle user bias. We propose an approach to model multi-round conversations within one session and take user information into account. Experimental results on a dataset from a real-world industrial customer service chatbot Alime demonstrate the good performance of our proposed model.

Nowadays, it is a heated topic for many industries to build automatic question-answering (QA) systems. A key solution to these QA systems is to retrieve from a QA knowledge base the most similar question of a given question, which can be reformulated as a paraphrase identification (PI) or a natural language inference (NLI) problem. However, most existing models for PI and NLI have at least two problems: They rely on a large amount of labeled data, which is not always available in real scenarios, and they may not be efficient for industrial applications. In this paper, we study transfer learning for the PI and NLI problems, aiming to propose a general framework, which can effectively and efficiently adapt the shared knowledge learned from a resource-rich source domain to a resource-poor target domain. Specifically, since most existing transfer learning methods only focus on learning a shared feature space across domains while ignoring the relationship between the source and target domains, we propose to simultaneously learn shared representations and domain relationships in a unified framework. Furthermore, we propose an efficient and effective hybrid model by combining a sentence encoding-based method and a sentence interaction-based method as our base model. Extensive experiments on both paraphrase identification and natural language inference demonstrate that our base model is efficient and has promising performance compared to the competing models, and our transfer learning method can help to significantly boost the performance. Further analysis shows that the inter-domain and intra-domain relationship captured by our model are insightful. Last but not least, we deploy our transfer learning model for PI into our online chatbot system, which can bring in significant improvements over our existing system. Finally, we launch our new system on the chatbot platform Eva in our E-commerce site AliExpress.

In microblogging, hashtags are used to be topical markers, and they are adopted by users that contribute similar content or express a related idea. However, hashtags are created in a free style and there is no domain category information about them, which make users hard to get access to organized hashtag presentation. In this paper, we propose an approach that classifies hashtags with news categories, and then carry out a domain-sensitive popularity ranking to get hot hashtags in each domain. The proposed approach first trains a domain classification model with news content and news category information, then detects microblogs related to a hashtag to be its representative text, based on which we can classify this hashtag with a domain. Finally, we calculate the domain-sensitive popularity of each hashtag with multiple factors, to get most hotly discussed hashtags in each domain. Preliminary experimental results on a dataset from Sina Weibo, one of the largest Chinese microblogging websites, show usefulness of the proposed approach on describing hashtags.

Over the last few years, microblogging is increasingly becoming an important platform for users to acquire information and publish some reviews or personal status. In microblogging, hash tags mean some topic words between two '#', such as some social events or some hot topics. Hash tags can highlight the topic of tweets, and make tweets be easily searched and understood by others. Therefore, many users like adding hash tags for their tweets. Existing hash tag recommendation methods always consider semantic similarity between hash tags and tweets as the only key factor. However, the hash tags' user acceptance degree and development tendency are two important factors for evaluate the recommendation probability of them. In this paper, we propose a model for recommending some related hash tags for users to choose one or more of them as content added into forthcoming tweets. The above three factors have been considered to complete this task, which are the semantic similarity between a hash tag and a tweet, the user acceptance degree of the hash tag, and the development tendency of the hash tag. Experimental results on a dataset from Sina Weibo, one of the largest Chinese microblogging websites, show usefulness of the proposed model for recommending hash tags to forthcoming tweets.

Due to its brief form and growing popularity, microblogging is becoming people's favorite choice for seeking information and expressing opinions. Messages received by a user mainly depend on whom the user follows. Thus, recommending users with similar interests may improve the experience quality for information receiving. Since messages posted by microblogging users reflect their interests, and the keywords in the messages indicate their main focus to a large extent, we can discover users' preferences by analyzing the user-generated contents. Moreover, users' interests are not static, on the contrary, they change as time goes by. Based on such intuitions, in this paper, we propose a temporal-topic model to analyze users' possible behaviors and predict their potential friends in microblogging. The model learns users' latent preferences by extracting keywords on aggregated messages over a period of time via a topic model, and then the impact of time is considered to deal with interest drifts. The experimental results of friend recommendations on Sina Weibo, one of the most popular microblogging sites in China, have demonstrated the effectiveness of our model.

The rapid increasing popularity of micro-blogging has made it an important information seeking channel. By detecting recent popular topics from micro-blogging, we have opportunities to gain insights into internet hotspots. Generally, a topic's popularity is determined by two primary factors. One is how frequently a topic is discussed by users, and the other is how much influence those users have, since topics shown in the influential users' posts are more likely to attract others' attention. However, existing approaches interpret a topic's popularity with only the number of keywords related to it, which neglect the importance of the user influence to information diffusion in micro-blogging. In this paper, drawing upon the Cognitive Authority Theory and Social Network Theory, we propose a novel model that detects the most popular topics in micro-blogging with a user interest-based method. The proposed model first constructs a topic graph according to users' interests and their following relationship, and then calculates the topics' popularity with a link-based ranking algorithm. The popular topics detected by the method can reflect the relationship among users' interests, and the topics in the posts of influential users can be highlighted. Experimental results on the data of Twitter, a well-known and feature-rich micro-blogging service, show that the proposed method is effective in popular topic discovery.