Large language models (LLMs) deliver impressive results but face challenges from increasing model sizes and computational costs.Structured pruning reduces model size and speeds up inference but often causes uneven degradation across domains, leading to biased performance.To address this, we propose DR-Pruning, a method that dynamically adjusts the data distribution during training to restore balanced performance across heterogeneous and multi-tasking data.Experiments in monolingual and multilingual settings show that DR-Pruning surpasses similarly sized models in both pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning.Further analysis demonstrates the robustness of DRPruning towards various domains and distribution shifts.Furthermore, DRPruning can determine optimal reference losses and data ratios automatically, suggesting potential for broader applications.

Despite the impressive performance of large language models (LLMs) in general domains, they often underperform in specialized domains.Existing approaches typically rely on data synthesis methods and yield promising results by using unlabeled data to capture domain-specific features.However, these methods either incur high computational costs or suffer from performance limitations, while also demonstrating insufficient generalization across different tasks.To address these challenges, we propose AQuilt, a framework for constructing instruction-tuning data for any specialized domains from corresponding unlabeled data, including Answer, Question, Unlabeled data, Inspection, Logic, and Task type.By incorporating logic and inspection, we encourage reasoning processes and self-inspection to enhance model performance.Moreover, customizable task instructions enable high-quality data generation for any task.As a result, we construct a dataset of 703k examples to train a powerful data synthesis model.Experiments show that AQuilt is comparable to DeepSeek-V3 while utilizing just 17% of the production cost.Further analysis demonstrates that our generated data exhibits higher relevance to downstream tasks.

Consistency learning (CL) has proven to be a valuable technique for improving the robustness of models in conditional sentence generation (CSG) tasks by ensuring stable predictions across various input data forms.However, models augmented with CL often face challenges in optimizing consistency features, which can detract from their efficiency and effectiveness.To address these challenges, we introduce Curriculum Consistency Learning (CCL), a novel strategy that guides models to learn consistency in alignment with their current capacity to differentiate between features.CCL is designed around the inherent aspects of CL-related losses, promoting task independence and simplifying implementation.Implemented across four representative CSG tasks, including instruction tuning (IT) for large language models and machine translation (MT) in three modalities (text, speech, and vision), CCL demonstrates marked improvements.Specifically, it delivers +2.0 average accuracy point improvement compared with vanilla IT and an average increase of +0.7 in COMET scores over traditional CL methods in MT tasks.Our comprehensive analysis further indicates that models utilizing CCL are particularly adept at managing complex instances, showcasing the effectiveness and efficiency of CCL in improving CSG models.Code and scripts are available at https://github.com/xinxinxing/ Curriculum-Consistency-Learning.

Despite their remarkable abilities in various tasks, large language models (LLMs) still struggle with real-time information (e.g., new facts and terms) due to the knowledge cutoff in their development process. However, existing benchmarks focus on outdated content and limited fields, facing difficulties in real-time updating and leaving new terms unexplored. To address this problem, we propose an adaptive benchmark, NewTerm, for real-time evaluation of new terms. We design a highly automated construction method to ensure high-quality benchmark construction with minimal human effort, allowing flexible updates for real-time information. Empirical results on various LLMs demonstrate over 20% performance reduction caused by new terms. Additionally, while updates to the knowledge cutoff of LLMs can cover some of the new terms, they are unable to generalize to more distant new terms. We also analyze which types of terms are more challenging and why LLMs struggle with new terms, paving the way for future research. Finally, we construct NewTerm 2022 and 2023 to evaluate the new terms updated each year and will continue updating annually. The benchmark and codes can be found at https://anonymous.4open.science/r/NewTerms.

Simultaneous machine translation (SiMT) is usually done via sequence-level knowledge distillation (Seq-KD) from a full-sentence neural machine translation (NMT) model. However, there is still a significant performance gap between NMT and SiMT. In this work, we propose to leverage monolingual data to improve SiMT, which trains a SiMT student on the combination of bilingual data and external monolingual data distilled by Seq-KD. Preliminary experiments on En-Zh and En-Ja news domain corpora demonstrate that monolingual data can significantly improve translation quality (e.g., +3.15 BLEU on En-Zh). Inspired by the behavior of human simultaneous interpreters, we propose a novel monolingual sampling strategy for SiMT, considering both chunk length and monotonicity. Experimental results show that our sampling strategy consistently outperforms the random sampling strategy (and other conventional typical NMT monolingual sampling strategies) by avoiding the key problem of SiMT -- hallucination, and has better scalability. We achieve +0.72 BLEU improvements on average against random sampling on En-Zh and En-Ja. Data and codes can be found at https://github.com/hexuandeng/Mono4SiMT.