Xuemiao Zhang, Can Ren, Chengying Tu, Rongxiang Weng, Hongfei Yan, Jingang Wang, Xunliang Cai. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

The performance of Large Language Models (LLMs) is intrinsically linked to the quality of its training data. Although several studies have proposed methods for high-quality data selection, they do not consider the importance of knowledge richness in text corpora. In this paper, we propose a novel and gradient-free High-Knowledge Scorer (HKS) to select high-quality data from the dimension of knowledge, to alleviate the problem of knowledge scarcity in the pre-trained corpus. We propose a comprehensive multi-domain knowledge element pool and introduce knowledge density and coverage as metrics to assess the knowledge content of the text. Based on this, we propose a comprehensive knowledge scorer to select data with intensive knowledge, which can also be utilized for domain-specific high-knowledge data selection by restricting knowledge elements to the specific domain. We train models on a high-knowledge bilingual dataset, and experimental results demonstrate that our scorer improves the model's performance in knowledge-intensive and general comprehension tasks, and is effective in enhancing both the generic and domain-specific capabilities of the model.

Selecting high-quality data can improve the pretraining efficiency of large language models (LLMs).Existing methods generally rely on heuristic techniques or single quality signals, limiting their ability to evaluate data quality comprehensively.In this work, we propose FIRE, a flexible and scalable framework for integrating multiple data quality raters, which allows for a comprehensive assessment of data quality across various dimensions.FIRE aligns multiple quality signals into a unified space, and integrates diverse data quality raters to provide a comprehensive quality signal for each data point.Further, we introduce a progressive data selection scheme based on FIRE that iteratively refines the selection of high-quality data points.Extensive experiments show that FIRE outperforms other data selection methods and significantly boosts pretrained model performance across a wide range of downstream tasks, while requiring less than 37.5% tokens needed by the Random baseline to reach the target performance.

Semi-supervised learning (SSL) is a promising approach to dealing with the problem of insufficient medical labeled data. Recent methods grouped into paradigms of consistency regularization and pseudo-labeling have outstanding performances in processing images but achieve limited improvements in processing medical texts. This paper proposes MedMatch, a novel semi-supervised learning (SSL) method addressing the challenges of medical text processing due to its discrete nature and lack of quality text augmentation. MedMatch can detect abnormal model states from anomalous data, reducing their impact while improving consistency regularization effectiveness. It also optimizes low entropy from pseudo-label learning. We introduce robust medical text augmentation pipelines to increase text view divergence, enhancing model learning. Comparative experiments show MedMatch significantly outperforms previous SSL methods on two major medical benchmarks.

Medical Image Report Generation aims to automatically generate medical reports based on radiology images, thus freeing radiologists from the tedious task of writing reports. Generating report texts that match the content of a given medical image, focus on local anomalies, and fluently conform to professional reporting norms presents the challenge of linking visual patterns to informative human-language descriptions. In this paper, we propose a novel generative adversarial framework (MIRGAN) to guide the generator to generate medical reports that are indistinguishable from those written by professional radiologists. MIRGAN introduces a multimodal discriminator to evaluate the performance of the report generator on the comprehensive objective containing three sub-objectives when generating each token. MIRGAN then uses it as the reward in reinforcement learning to guide the generator to optimize toward the desired objective in real time. We conduct sufficient experiments to evaluate MIRGAN on two widely used datasets for chest-related diseases. The experimental results show that MIRGAN can significantly improve the performance of the generator on most metrics.

Medical Image Report Generation aims to automate the creation of medical reports from radiology images, alleviating the burden on radiologists. The challenge lies in producing text that aligns with the image content, highlights local anomalies, and adheres to professional norms. In this study, we restructure this task as a multi-objective optimization problem and introduce a novel adversarial training framework, MORGAN, with multiple discriminators to guide the generation of reports akin to those written by professional radiologists. MORGAN employs three discriminators to assess the alignment of image-text, focus on local anomalies, and adherence to medical norms when generating each token. These assessments serve as reinforcement learning rewards, guiding the generator to optimize these objectives in real time. We extensively test MORGAN on two prevalent chest-related disease datasets. The results demonstrate that MORGAN significantly enhances the generator’s performance on most metrics.

Weakly supervised machine reading comprehension (MRC) task is practical and promising for its easily available and massive training data, but inevitablely introduces noise.Existing related methods usually incorporate extra submodels to help filter noise before the noisy data is input to main models.However, these multistage methods often make training difficult, and the qualities of submodels are hard to be controlled.In this paper, we first explore and analyze the essential characteristics of noise from the perspective of loss distribution, and find that in the early stage of training, noisy samples usually lead to significantly larger loss values than clean ones.Based on the observation, we propose a hierarchical loss correction strategy to avoid fitting noise and enhance clean supervision signals, including using an unsupervisedly fitted Gaussian mixture model to calculate the weight factors for all losses to correct the loss distribution, and employ a hard bootstrapping loss to modify loss function.Experimental results on different weakly supervised MRC datasets show that the proposed methods can help improve models significantly.

Naive neural dialogue generation models tend to produce repetitive and dull utterances. The promising adversarial models train the generator against a well-designed discriminator to push it to improve towards the expected direction. However, assessing dialogues requires consideration of many aspects of linguistics, which are difficult to be fully covered by a single discriminator. To address it, we reframe the dialogue generation task as a multi-objective optimization problem and propose a novel adversarial dialogue generation framework with multiple discriminators that excel in different objectives for multiple linguistic aspects, called AMPGAN, whose feasibility is proved by theoretical derivations. Moreover, we design an adaptively adjusted sampling distribution to balance the discriminators and promote the overall improvement of the generator by continuing to focus on these objectives that the generator is not performing well relatively. Experimental results on two real-world datasets show a significant improvement over the baselines.

Knowledge bases (KBs) provide a large amount of structured information for entities and relations, which are successfully leveraged in many natural language processing tasks. However, distantly supervised relation extraction only utilizes KBs to automatically generate datasets, while ignoring the b ackground information in KBs during the relation extraction process. We herein propose a knowledge-embodied attention that leverages knowledge information in KBs to reduce the impact of noisy data for distantly supervised relation extraction. Specifically, we pre-train distributed representations of KBs with the knowledge representation learning (KRL) model, and subsequently incorporate them into relation extraction to learn sentence-level attention weights. The experimental results demonstrate that our approach outperforms all baselines, thus indicating that we can focus our attention on valid data by leveraging background information in KBs.