Transformer-based large language models (LLMs) excel in natural language processing tasks by capturing long-range dependencies through self-attention mechanisms. However, long-context modeling faces significant computational inefficiencies due to redundant attention computations: while attention weights are often sparse, all tokens consume equal computational resources. In this paper, we reformulate traditional probabilistic sequence modeling as a supervised learning task, enabling the separation of relevant and irrelevant tokens and providing a clearer understanding of redundancy. Based on this reformulation, we theoretically analyze attention sparsity, revealing that only a few tokens significantly contribute to predictions. Building on this, we formulate attention optimization as a linear coding problem and propose a group coding strategy, theoretically showing its ability to improve robustness against random noise and enhance learning efficiency. Motivated by this, we propose Dynamic Group Attention (DGA), which leverages the group coding to explicitly reduce redundancy by aggregating less important tokens during attention computation. Empirical results show that our DGA significantly reduces computational costs while maintaining competitive performance.

Open-domain dialogue systems aim to generate natural and engaging conversations, providing significant practical value in real applications such as social robotics and personal assistants. The advent of large language models (LLMs) has greatly advanced this field by improving context understanding and conversational fluency. However, existing LLM-based dialogue systems often fall short in proactively understanding the user's chatting preferences and guiding conversations toward user-centered topics. This lack of user-oriented proactivity can lead users to feel unappreciated, reducing their satisfaction and willingness to continue the conversation in human-computer interactions. To address this issue, we propose a User-oriented Proactive Chatbot (UPC) to enhance the user-oriented proactivity. Specifically, we first construct a critic to evaluate this proactivity inspired by the LLM-as-a-judge strategy. Given the scarcity of high-quality training data, we then employ the critic to guide dialogues between the chatbot and user agents, generating a corpus with enhanced user-oriented proactivity. To ensure the diversity of the user backgrounds, we introduce the ISCO-800, a diverse user background dataset for constructing user agents. Moreover, considering the communication difficulty varies among users, we propose an iterative curriculum learning method that trains the chatbot from easy-to-communicate users to more challenging ones, thereby gradually enhancing its performance. Experiments demonstrate that our proposed training method is applicable to different LLMs, improving user-oriented proactivity and attractiveness in open-domain dialogues. Code and appendix are available at github.com/wang678/LLM-UPC.

Qianyue Wang, Jinwu Hu, Zhengping Li, Yufeng Wang, Daiyuan Li, Yu Hu, Mingkui Tan. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Existing warning methods are hindered by the ability to extract subtle early-stage blade icing features from wind turbine (WT) monitoring data, resulting in delayed or unreliable alerts. To address this challenge, this paper proposes a novel diffusion-based normal WT behavior model, Conditional Time Series Denoising Diffusion (CTSDD), and develops an automated predictive fault warning system accordingly. CTSDD restores normal monitoring data from noise, which encodes conditional information generated by aHybrid Maskingstrategy to guide the restoration process. Its conditional encoding and noise prediction networks adopt a Transformer-like symmetric structure, integrating Dual-Attention and bidirectional long short-term memory (BiLSTM) modules to enhance feature extraction for multidimensional time series data. During the early stages of blade icing, deviations from learned normal patterns result in a noticeable decline in wind power (WP) restoration accuracy. Leveraging these deviations, this study proposes a composite scoring algorithm and a thresholding scheme to evaluate WT status and enable timely early warnings. Comparative experiments validate CTSDD’s superior restoration performance, with its system triggering the earliest and most stable warnings—approximately five hours before fault escalation. Ablation studies further confirm the contributions of Dual-Attention and BiLSTM modules. The proposed system offers an intelligent and effective solution to enhance the safety and reliability of WT operations.