Yitong Wang, Xue Han, WenChun Gao, Qian Hu, Jiahui Wang, Ziqing Wang, Lijun Mei, Junlan Feng. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Large Language Models (LLMs) are often English-centric due to the disproportionate distribution of languages in their pre-training data. Enhancing non-English language capabilities through post-pretraining often results in catastrophic forgetting of high-resource languages. Previous methods either achieve good expansion with severe forgetting or slight forgetting with poor expansion, indicating the challenge of balancing language expansion while preventing forgetting. In this paper, we propose a method called MoE-LPR (Mixture-of-Experts with Language Priors Routing) to alleviate this problem. MoE-LPR employs a two-stage training approach to enhance the multilingual capability. First, the model is post-pretrained into a Mixture-of-Experts(MoE) architecture by upcycling, where all the original parameters are frozen and new experts are added. In this stage, we focus improving the ability on expanded languages, without using any original language data. Then, the model reviews the knowledge of the original languages with replay data amounting to less than 1% of post-pretraining, where we incorporate language priors routing to better recover the abilities of the original languages. Evaluations on multiple benchmarks show that MoE-LPR outperforms other post-pretraining methods. Freezing original parameters preserves original language knowledge while adding new experts preserves the learning ability. Reviewing with LPR enables effective utilization of multilingual knowledge within the parameters. Additionally, the MoE architecture maintains the same inference overhead while increasing total model parameters. Extensive experiments demonstrate MoE-LPR’s effectiveness in improving expanded languages and preserving original language proficiency with superior scalability.

Cross-lingual context retrieval (extracting contextual information in one language based on requests in another) is a fundamental aspect of cross-lingual alignment, but the performance and mechanism of it for large language models (LLMs) remains unclear.In this paper, we evaluate the cross-lingual context retrieval of over 40 LLMs across 12 languages, using cross-lingual machine reading comprehension (xMRC) as a representative scenario.Our results show that post-trained open LLMs show strong cross-lingual context retrieval ability, comparable to closed-source LLMs such as GPT-4o, and their estimated oracle performances greatly improve after posttraining.Our mechanism analysis shows that the cross-lingual context retrieval process can be divided into two main phases: question encoding and answer retrieval, which are formed in pre-training and post-training respectively.The phasing stability correlates with xMRC performance, and the xMRC bottleneck lies at the last model layers in the second phase, where the effect of post-training can be evidently observed.Our results also indicate that largerscale pretraining cannot improve the xMRC performance.Instead, larger LLMs need further multilingual post-training to fully unlock their cross-lingual context retrieval potential.

Large-scale pre-trained language models (PLMs) require significant computational resources to train from scratch on large volumes of data. But in the real world, emerging data from diverse sources may not be initially available for pre-training. Recent studies on lifelong learning have tried to solve this problem by exploring the use of model growth techniques to effectively incorporate new knowledge without the need for complete re-training. However, model growth approaches utilized have issues with growth operators that do not ensure strict function preservation or growth schedules that only include a few growth dimensions, reducing lifelong learning's effect. Furthermore, existing approaches often assume that emerging data has the same distribution as pre-training data, causing catastrophic forgetting of previously acquired knowledge. To address the aforementioned issues, we introduce LOIRE, a framework for lifelong learning that enables PLMs to effectively grow their capacity using incremental data. LOIRE employs growth operators for all feasible dimensions and a growth schedule to generate the optimal expansion sequence in the field of lifelong learning. Specifically, we present a novel plug-in layer growth operator with residual connections that skip the newly added layer during initial training while ensuring function preservation. We additionally propose an iterative distillation strategy for LOIRE that allows an intermediate model in the growth stages to switch between being a student and a teacher, reducing catastrophic forgetting during growth. Experiments show that LOIRE can reduce computational expenses by an average of 29.22\% while retaining equivalent or better downstream performance.

Peng Hu, Sizhe Liu, Changjiang Gao, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Shujian Huang. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Shimao Zhang, Changjiang Gao, Wenhao Zhu, Jiajun Chen, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Shujian Huang. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Active learning (AL) aims to reduce labeling costs by selecting the most valuable samples to annotate from a set of unlabeled data. However, recognizing these samples is particularly challenging in multi-label text classification tasks due to the high dimensionality but sparseness of label spaces. Existing AL techniques either fail to sufficiently capture label correlations, resulting in label imbalance in the selected samples, or suffer significant computing costs when analyzing the informative potential of unlabeled samples across all labels. Facing these challenges, we propose an efficient two-stage sample acquisition strategy for multi-label active learning, called ALMuLa-mix. For saving the computational costs, ALMuLa-mix first attempts to identify unlabeled samples with novel features by employing a time-efficient feature-mixing method in conjunction with label correlations. Regarding the label imbalance, ALMuLa-mix then leverages the minority class in the labeled set to select a small batch of candidate unlabeled samples with greater inter-class diversity from the candidates with novel features. Experimental results on publicly available datasets show that ALMuLa-mix is superior to other strong baselines for handling multi-label text classification tasks.

To improve the robustness of pre-trained language models (PLMs), previous studies have focused more on how to efficiently obtain adversarial samples with similar semantics, but less attention has been paid to the perturbed samples that change the gold label. Therefore, to fully perceive the effects of these different types of small perturbations on robustness, we propose a RObust Self-supervised leArning (ROSA) method, which incorporates different types of perturbed samples and the robustness improvements into a unified framework. Subsequently, to implement ROSA, a perturbed sample generation strategy supported by the large language models (LLMs) is proposed, which adaptively controls the generation process based on the fine-grained similarity information among the training samples. The experimental results demonstrate the remarkable performance of our ROSA.

The utilization of web social media for depression detection has been proven effective in recent years since the multimedia signal on web can reflect users’ emotions, feelings, and personality traits in advance. However, most earlier studies simply used users’ submitted words or user profiles to predict depression risk. The implicit information accessible in users’ posted images, which can be effective in depression detection, still remains unexplored. In this paper, an implicit and explicit multi-modal feature fusion (IEMFF) model is proposed for depression detection. We successfully make the implicit information inherent in users’ posted images explicit and further incorporate such explicit features with the textual features directly extracted from user-posted texts. A multi-modal feature fusion approach is applied for depression detection. Extensive experiments have been conducted on public Twitter datasets. Experimental results show that our approach has achieved state-of-the-art performance for depression detection.

Fine-grained address entity recognition (FGAER) from multi-turn spoken dialogues is particularly challenging.The major reason lies in that a full address is often formed through a conversation process.Different parts of an address are distributed through multiple turns of a dialogue with spoken noises.It is nontrivial to extract by turn and combine them.This challenge has not been well emphasized by main-stream entity extraction algorithms.To address this issue, we propose in this paper a logic-guided fine-grained address recognition method (Log-FGAER), where we formulate the address hierarchy relationship as the logic rule and softly apply it in a probabilistic manner to improve the accuracy of FGAER.In addition, we provide an ontology-based data augmentation methodology that employs ChatGPT to augment a spoken dialogue dataset with labeled address entities.Experiments are conducted using datasets generated by the proposed data augmentation technique and derived from real-world scenarios.The results of the experiment demonstrate the efficacy of our proposal.Where are you in Suzhou city, Jiangsu Prov? 您现在在江苏 苏州市 的哪里？ I'm in Wujiang District, Shengze Town in Wujiang District.

Previous contrastive learning methods for sentence representations often focus on insensitive transformations to produce positive pairs, but neglect the role of sensitive transformations that are harmful to semantic representations. Therefore, we propose an Equivariant Self-Contrastive Learning (ESCL) method to make full use of sensitive transformations, which encourages the learned representations to be sensitive to certain types of transformations with an additional equivariant learning task. Meanwhile, in order to improve practicability and generality, ESCL simplifies the implementations of traditional equivariant contrastive methods to share model parameters from the perspective of multi-task learning. We evaluate our ESCL on semantic textual similarity tasks. The proposed method achieves better results while using fewer learning parameters compared to previous methods.

Dialog Router is a general paradigm for human-bot symbiosis dialog systems to provide friendly customer care service. It is equipped with a multi-task learning model to automatically capture the underlying correlation between multiple related tasks, i.e. dialog classification and regression, and greatly reduce human labor work for system customization, which improves the accuracy of dialog transition. In addition, for learning the multi-task model, the training data and labels are easy to collect from human-to-human historical dialog logs, and the Dialog Router can be easily integrated into the majority of existing dialog systems by calling general APIs. We conduct experiments on real-world datasets for dialog classification and regression. The results show that our model achieves improvements on both tasks, which benefits the dialog transition application. The demo illustrates our method’s effectiveness in a real customer care service.

Automatic business process discovery from textual process documentation is highly desirable to reduce the time and cost of Business Process Management (BPM) implementation in organizations. However, existing automatic process discovery approaches mainly focus on identifying activities out of the documentations. Deriving the hierarchical structural relationships between activities, which is important in the whole process discovery scope, requires great human labeling effort and is still a challenge. Facing this challenge, we propose to retrieve the latent hierarchical structure present in the textual process documentation by building a neural network without any extra human-labeled knowledge. The proposed neural network leverages a novel recurrent architecture, Ordered Neurons LSTM (ON-LSTM), with a process-level language model objective. On the base of this, we provide an automatic business process service (A-BPS) which could support the goal of automatically discovering business processes out of the uploaded documentation and generate Business Process Model and Notation (BPMN) scripts to represent the discovered process model through a service ecosystem. Experimental results show that 58.76% of the hierarchical structures could be correctly retrieved. A-BPS could generate 32% of the whole BPMN process model with average execution time 5-8 minutes. The time benefit of A-BPS is up to 40% reduction.

The use of social media runs through our lives, and users' emotions are also affected by it. Previous studies have reported social organizations and psychologists using social media to find depressed patients. However, due to the variety of content published by users, it isn't effortless for the system to consider the text, image, and even the hidden information behind the image. To address this problem, we proposed a new system for social media screening of depressed patients named BlueMemo. We collected real-time posts from Twitter. Based on the posts, learned text features, image features, and visual attributes were extracted as three modalities and were fed into a multi-modal fusion and classification model to implement our system. The proposed BlueMemo has the power to help physicians and clinicians quickly and accurately identify users at potential risk for depression.

Natural Language Processing (NLP) is a key technique to automate Business Process Management (BPM) at different levels. The performance of existing NLP based BPM methods suffer from the limited accuracy of Part of Speech (POS) tagging, which is a key step in NLP pipelines. Note that the performance of POS tagging highly depends on the domain of annotated training data. However, most state-of-the-art POS taggers are trained from corpus in newswire domain which usually have different syntax features with business process description (BPD). The syntax features of BPD domain include usually starting with an imperative verb and containing numerous out-of-vocabulary (OOV) words. In this paper, we propose a novel POS tagging framework to tackle these problems. The main idea is that syntax feature of starting with imperative verb could be studied by enhancing the proportion of correctly POS-annotated imperative sentences in the training data. The trained POS tagger could reduce the overall POS tagging error by nearly 12% compared with newswire trained POS tagger. For verbs which are key words in BPD, the tagging precision could be increased by 27%. The lexical ambiguity caused by OOV words is solved by extracting local contextual knowledge out of images which are attached to help users understand the process better. Experimental results show that the overall POS tagging accuracy could be increased by nearly 10% with contextual OOV knowledge.

Interference mitigation is a major challenge in deploying a two-tier small cell network, where small cells are deployed underlaying a central macrocell and share the same spectrum with the macrocell. In this paper, we develop a new decentralized power control solution for interference mitigation in a two-tier small cell network, from a game theoretic perspective. We aim to maximize the number of small cell user (SU) transmissions that can be admitted in the network while satisfying the signal-to- interference-noise ratio (SINR) constraints of both transmitting SUs and the macrocell user (MU). We formulate the problem of power control for SUs as a game with common utility. The Nash equilibria of the game are investigated. We then propose a learning automata based distributed discrete power control algorithm with which the SUs can learn from their action-reward histories and adjust their transmit powers towards a NE point. Simulation results show the proposed algorithm achieves higher number of SU transmissions that can be admitted compared with existing schemes in the literature.

Cellular relay networks prove to be a cost-effective approach that offers significant performance benefits in coverage extension, cell-edge throughput enhancement and increased spectral efficiency. Radio resource management for cellular relay networks, where the network topology is highly dynamic, is particularly challenging. The highly dynamic topology may be caused by an ever increasing number of users accessing Internet and other multimedia services using mobile devices carried by pedestrians, in a train or in vehicles. This paper studies the subcarrier allocation problem in highly dynamic cellular relay networks with an objective to maximize the overall throughput. This optimization problem is formulated and solved as an expectation maximization problem. Statistical multiplexing gain is explicitly explored to further improve the channel utilization and spectral efficiency. Numerical evaluations are performed which demonstrate that the proposed scheme offers higher system throughput and improved radio resource utilization compared with existing schemes.

Small cell cluster is a new paradigm to extend the usage of small cells from residential environment to large indoor or outdoor areas. However, coverage optimization is a challenge due to the ad-hoc deployment and plug-and-play feature of small cells. This paper considers decentralized self-optimization network (SON) architecture of small cell cluster and proposes distributed coverage optimization algorithm using game theory (DGT). A non-cooperate game is modeled to tune the Tx power of each small cell with a net utility function considering both gain of throughput and punishment of interference. Nash Equilibrium (NE) is proved to be existed in the game and a power update scheme is proposed which converges to the NE. Simulation results show that DGT can significantly improve throughput as well as coverage ratio with only several iterations. Compared with centralized algorithm such as modified particle swarm optimization (MPSO) and simulated annealing (SA), DGT algorithm reaches higher network throughput, uses less iteration and keeps considerable coverage ratio.

Due to the cooperation diversity, Coordinated MultiPoint (CoMP) transmission may potentially improve energy efficiency of cellular networks. The main purpose of this paper is to analyze this potential. By employing CoMP, there are two schemes to improve energy efficiency; one is to decrease the transmission power of base stations (CDTP) without changing cells' size and re-deploying base stations (BSs); another one is to reduce the number of BSs by increasing coverage area (CICA). In this paper, the coverage gain brought by CoMP technology given the guaranteed coverage ratio performance is firstly performed. Based on that, the energy efficiency gain of CDTP and CICA is investigated. It showed that comparing to the scheme of CDTP which is nearly no potential to improve energy efficiency, CICA can improve energy efficiency by more than 15%. This result could provide references for operators when carrying out new cellular deployments.

This paper focuses on the energy efficiency of two-stage cooperative multicast with CPC (CP combining). Aiming to provide a coverage of 95%, cooperative multicast with SC (selective combining) has been analytically investigated and a power saving of 43% can be achieved compared with conventional one-stage multicast [9]. However, CP combining which is more practical can provide a stronger received signal than SC and thus is expected to be able to provide further energy saving. Assuming high user density, analysis is carried out on the coverage performance of OFDM based two-stage cooperative multicast with CPC, which is decided by the BS (base station) transmission power at the 1st stage and the number and locations of MRs (mobile relays) at the 2nd stage. Given a fixed coverage, it is difficult to obtain the BS transmission power and MR scheme that provide the optimal total power consumption. Therefore, two sub-optimal schemes are proposed, i.e., BS coordination and MR coordination, which are based on the schemes for cooperative multicast with SC [9] and reduce the BS transmission power and the number of MRs, respectively. Numerical results show that with BS and MR coordination, cooperative multicast with CPC can save 9% and 17% compared to that with SC, respectively. Moreover, MR coordination provides more uniform coverage than BS coordination. Simulations are carried out to verify the numerical results. It is shown that MR coordination provides better coverage than BS coordination for all investigated user densities and the performance gap increases as the density decrease.

Coverage optimization is a main challenge for femtocell clusters which are considered to be a promising solution to provide seamless cellular coverage for large indoor areas. Although particle swarm optimization (PSO) can be employed to solve the coverage optimization of femtocell clusters, it reduces to single particle swarm optimization (SPSO) when only one femtocell cluster is considered and can only find the local optimum solution. In this paper, a modified PSO (MPSO) algorithm is proposed, which employs a heuristic power control scheme to guide the algorithm to search for the global optimum solution. Simulation results show that MPSO significantly outperforms SPSO and fixed power scheme with low complexity. Moreover, MPSO can converge rapidly and is suitable for online coverage optimization.

With the rapid development of high-speed wireless communication technology, more novel and different wireless networks are emerging. The co-existence of multiple wireless networks has made the wireless network environment complex and heterogeneous. Traffic flow characteristics study and effective traffic prediction algorithm selection are very important for optimal allocation of network resources, network protocol design and improvement of service quality in this heterogeneous wireless network. This paper uses non-statistical method which is based on chaos theory to make analysis of heterogeneous wireless network traffic flow characteristics. On that basis, a traffic model which uses phase space reconstruction algorithm on multivariate traffic time series is proposed and used for traffic prediction. Simulation results show that the proposed traffic prediction model can achieve better prediction performance.

In order to maintain sufficient mobile communication network capacity for value added services, 3GPP has recently introduced an architecture called ”SIPTO”(Selected IP Traffic Offload) to offload selected mobile IP traffic from the core network. This brings new challenge to on line traffic classification scheme needed to figure out the traffic which should be offloaded in real time. Although traffic classification methodologies have already been investigated in wired IP network, they are not applicable in mobile environment where high bit error rates (BER) and temporary disconnections are observed due to hostile wireless channel conditions. This paper proposes a maximum entropy based IP-traffic classification scheme (METCS) to address on line traffic classification problems in mobile communication networks. METCS extracts the application layer payload pattern using randomly arrived packets instead of the first few sequential packets of an application flow. Simulation results show that METCS outperforms existing methods by offering average 5%-8% improvement in classification accuracy with about 60% time.