Xiangchi Yuan, Dachuan Shi, Chunhui Zhang, Zheyuan Liu, Shenglong Yao, Soroush Vosoughi, Wenke Lee. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

As scaled language models (LMs) approach human-level reasoning capabilities, selfimprovement emerges as a solution to synthesizing high-quality data corpus.While previous research has identified model collapse as a risk in self-improvement, where model outputs become increasingly deterministic, we discover a more fundamental challenge: the superficial self-improved reasoners phenomenon.In particular, our analysis reveals that even when LMs show improved in-domain (ID) reasoning accuracy, they actually compromise their generalized reasoning capabilities on out-of-domain (OOD) tasks due to memorization rather than genuine learning.Through a systematic investigation of LM architecture, we discover that during self-improvement, LM weight updates are concentrated in less reasoning-critical layers, leading to superficial learning.To address this, we propose Iterative Model Merging (IMM), a method that strategically combines weights from original and self-improved models to preserve generalization while incorporating genuine reasoning improvements.Our approach effectively mitigates both LM collapse and superficial learning, moving towards more stable self-improving systems.Code is available 1 .

Recent advancements in Large Language Models (LLMs) have spurred interest in numerous applications requiring robust long-range capabilities, essential for processing extensive input contexts and continuously generating extended outputs. As sequence lengths increase, the number of Key-Value (KV) pairs in LLMs escalates, creating a significant efficiency bottleneck. In this paper, we propose a new KV cache optimization paradigm called LaCache, a training-free method for efficient and accurate generative inference of LLMs. LaCache enables LLMs to simultaneously address both of the critical challenges in long-range modeling: robust long-range capabilities and continuous generation without running out-of-memory (OOM). Specifically, LaCache integrates two key innovations: (1) a ladder-shaped KV cache pattern that stores KV pairs not only sequentially (left-to-right within each layer) but also across layers (from shallow to deep), providing an extended span for capturing long-range dependencies under a fixed storage budget, thereby boosting long-range capabilities; and (2) an iterative compaction mechanism that progressively compresses older caches, freeing up space for new tokens within a fixed cache size. This token distance-based dynamic compression enables more effective continuous generation under constrained cache budgets. Experiments across various tasks, benchmarks, and LLM models consistently validate LaCache’s effectiveness in enhancing LLMs’ long-range capabilities. Our code is available at https://github.com/GATECH-EIC/LaCache.

Entity Matching (EM) concerns identifying entities from different data sources that correspond to the same real-world object. It is widely used for product data integration in e-commerce, product classification, and inventory management, enabling the matching of duplicate product records with heterogeneous descriptions across various platforms and software systems. The standard EM solution consists of two steps: a blocking step to retrieve a subset of candidates and a pairwise matching step to classify whether the query entity matches each candidate. However, a significant challenge arises when pairwise matching fails to account for similar distractors within the candidate subset, often leading to false positive matches. This issue has been largely overlooked in prior work and existing benchmark datasets. In this study, we address this gap through three key aspects. First, we revisit the standard pairwise EM setting by recompiling existing benchmark datasets to include more hard negative (HN) candidates, which are semantically similar to corresponding query entities. We then evaluate state-of-the-art (SOTA) pairwise matchers on these recompiled datasets, revealing the limitations of the conventional pairwise EM approach under more challenging and realistic conditions. Second, we propose a selective EM approach that formulates EM as a listwise selection task, where the query entity is compared directly with the entire candidate set rather than evaluated through independent pairwise classifications. Accordingly, a new evaluation framework is introduced, including recompiled benchmark datasets and a new evaluation metric. Third, we propose a selective EM method Mistral4SelectEM, which fine-tunes a large language model for selective EM by structuring it into a Siamese network and fine-tuning it with a novel contrastive margin ranking loss (CMRL). It aims to enhance the model’s ability to distinguish true positives from semantically similar HNs. Extensive experiments demonstrate that our method outperforms SOTA pairwise EM approaches in both efficiency and performance across multiple benchmark datasets. The code and the recompiled entity matching benchmark datasets are publicly available at: https://github.com/quickhdsdc/LLM4EntityMatching .

Recent vision-language models have achieved tremendous advances. However, their computational costs are also escalating dramatically, making model acceleration exceedingly critical. To pursue more efficient vision-language Transformers, this paper introduces Cross-Guided Ensemble of Tokens (CrossGET), a general acceleration framework for vision-language Transformers. This framework adaptively combines tokens in real-time during inference, significantly reducing computational costs while maintaining high performance. CrossGET features two primary innovations: 1) Cross-Guided Matching and Ensemble. CrossGET leverages cross-modal guided token matching and ensemble to effectively utilize cross-modal information, achieving wider applicability across both modality-independent models, e.g., CLIP, and modality-dependent ones, e.g., BLIP2. 2) Complete-Graph Soft Matching. CrossGET introduces an algorithm for the token-matching mechanism, ensuring reliable matching results while facilitating parallelizability and high efficiency. Extensive experiments have been conducted on various vision-language tasks, such as image-text retrieval, visual reasoning, image captioning, and visual question answering. The performance on both classic multimodal architectures and emerging multimodal LLMs demonstrates the framework’s effectiveness and versatility. The code is available at https://github.com/sdc17/CrossGET.

Motivated by the transformative capabilities of large language models (LLMs) across various natural language tasks, there has been a growing demand to deploy these models effectively across diverse real-world applications and platforms. However, the challenge of efficiently deploying LLMs has become increasingly pronounced due to the varying application-specific performance requirements and the rapid evolution of computational platforms, which feature diverse resource constraints and deployment flows. These varying requirements necessitate LLMs that can adapt their structures (depth and width) for optimal efficiency across different platforms and application specifications. To address this critical gap, we propose AmoebaLLM, a novel framework designed to enable the instant derivation of LLM subnets of arbitrary shapes, which achieve the accuracy-efficiency frontier and can be extracted immediately after a one-time fine-tuning. In this way, AmoebaLLM significantly facilitates rapid deployment tailored to various platforms and applications. Specifically, AmoebaLLM integrates three innovative components: (1) a knowledge-preserving subnet selection strategy that features a dynamic-programming approach for depth shrinking and an importance-driven method for width shrinking; (2) a shape-aware mixture of LoRAs to mitigate gradient conflicts among subnets during fine-tuning; and (3) an in-place distillation scheme with loss-magnitude balancing as the fine-tuning objective. Extensive experiments validate that AmoebaLLM not only sets new standards in LLM adaptability but also successfully delivers subnets that achieve state-of-the-art trade-offs between accuracy and efficiency.

Real-world data contains a vast amount of multimodal information, among which vision and language are the two most representative modalities. Moreover, increasingly heavier models, e.g., Transformers, have attracted the attention of researchers to model compression. However, how to compress multimodal models, especially vison-language Transformers, is still under-explored. This paper proposes the Unified and Progressive Pruning (UPop) as a universal vison-language Transformer compression framework, which incorporates 1) unifiedly searching multimodal subnets in a continuous optimization space from the original model, which enables automatic assignment of pruning ratios among compressible modalities and structures; 2) progressively searching and retraining the subnet, which maintains convergence between the search and retrain to attain higher compression ratios. Experiments on various tasks, datasets, and model architectures demonstrate the effectiveness and versatility of the proposed UPop framework. The code is available at https://github.com/sdc17/UPop.

For medical image segmentation in a real scenario, the amount of accurate annotation data at the pixel level is typically small, which tends to cause an overfitting problem. This manuscript goes deep into the research of the Dropout algorithm, which is commonly used in neural networks to alleviate the overfitting problem. From the perspective of solving the co-adaptation problem, this manuscript explains the basic principles of the Dropout algorithm and discusses the existing limitations of its derivative methods. Furthermore, we propose a novel Heuristic Dropout algorithm to address these limitations. The proposed algorithm takes information entropy and variance as heuristic rules. It guides our algorithm to drop features suffering from co-adaptation problem more efficiently and thus can better alleviate the overfitting problem of small-scale medical image segmentation datasets. Experiments on medical image segmentation datasets and models show that the proposed algorithm significantly improves the performance of these models.

Second language (L2) English learners often find it difficult to improve their pronunciations due to the lack of expressive and personalized corrective feedback. In this paper, we present Pronunciation Teacher (PTeacher), a Computer-Aided Pronunciation Training (CAPT) system that provides personalized exaggerated audio-visual corrective feedback for mispronunciations. Though the effectiveness of exaggerated feedback has been demonstrated, it is still unclear how to define the appropriate degrees of exaggeration when interacting with individual learners. To fill in this gap, we interview 100 L2 English learners and 22 professional native teachers to understand their needs and experiences. Three critical metrics are proposed for both learners and teachers to identify the best exaggeration levels in both audio and visual modalities. Additionally, we incorporate the personalized dynamic feedback mechanism given the English proficiency of learners. Based on the obtained insights, a comprehensive interactive pronunciation training course is designed to help L2 learners rectify mispronunciations in a more perceptible, understandable, and discriminative manner. Extensive user studies demonstrate that our system significantly promotes the learners’ learning efficiency.

Deep learning models, especially U-Net and its derivate models, have been widely used in medical image segmentation. These approaches have achieved promising results in many medical image segmentation tasks with a limited number of training samples. We aim on enhancing medical image segmentation by using spatial continuity information in a proposed Multi-Encoder Parse-Decoder Network (MEPDNet) based on the fact that most of the medical images are sampled continuously. Sequential images are input into parameter shared encoders for getting feature maps, which are then fused by a fusion block. A V$\Lambda$-block is structured to parse the fused feature map to extract the hidden continuity information. The reconstructed feature map is fed into a decoder for generating segmentation masks. Experiments on three datasets show MEPDNet outperforms other state-of-the-art segmentation models while using the least parameters.