Dominik Glandorf, Peng Cui, Detmar Meurers, Mrinmaya Sachan. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Abstract Human evaluation is the gold standard for evaluating text generation models. However, it is expensive. In order to fit budgetary constraints, a random subset of the test data is often chosen in practice for human evaluation. However, randomly selected data may not accurately represent test performance, making this approach economically inefficient for model comparison. Thus, in this work, we develop and analyze a suite of selectors to get the most informative datapoints for human evaluation, taking the evaluation costs into account. We show that selectors based on variance in automated metric scores, diversity in model outputs, or Item Response Theory outperform random selection. We further develop an approach to distill these selectors to the scenario where the model outputs are not yet available. In particular, we introduce source-based estimators, which predict item usefulness for human evaluation just based on the source texts. We demonstrate the efficacy of our selectors in two common NLG tasks, machine translation and summarization, and show that only ∼70% of the test data is needed to produce the same evaluation result as the entire data.

Using questions in written text is an effective strategy to enhance readability.However, what makes an active reading question good, what the linguistic role of these questions is, and what is their impact on human reading remains understudied.We introduce GUIDINGQ, a dataset of 10K in-text questions from textbooks and scientific articles.By analyzing the dataset, we present a comprehensive understanding of the use, distribution, and linguistic characteristics of these questions.Then, we explore various approaches to generate such questions using language models.Our results highlight the importance of capturing inter-question relationships and the challenge of question position identification in generating these questions.Finally, we conduct a human study to understand the implication of such questions on reading comprehension.We find that the generated questions are of high quality and are almost as effective as human-written questions in terms of improving readers' memorization and comprehension.github.com/eth-lre/engage-your-readers Questions in titles:How do Philosophers arrive at truth?Is there no quantum form of Einstein Gravity?Why do house-hunting ants recruit in both directions?

Adaptive learning aims to provide customized educational activities (e.g., exercises) to address individual learning needs.However, manual construction and delivery of such activities is a laborious process.Thus, in this paper, we study a novel task of adaptive and personalized exercise generation for online language learning.To this end, we combine a knowledge tracing model that estimates each student's evolving knowledge states from their learning history and a controlled text generation model that generates exercise sentences based on the student's current estimated knowledge state and instructor requirements of desired properties (e.g., domain knowledge and difficulty).We train and evaluate our model on real-world learner interaction data from Duolingo and demonstrate that LMs guided by student states can generate superior exercises.Then, we discuss the potential use of our model in educational applications using various simulations.These simulations show that our model can adapt to students' individual abilities and can facilitate their learning efficiency by personalizing learning sequences.1

Neural-based summarization models suffer from the length limitation of text encoder.Long documents have to been truncated before they are sent to the model, which results in huge loss of summary-relevant contents.To address this issue, we propose the sliding selector network with dynamic memory for extractive summarization of long-form documents, which employs a sliding window to extract summary sentences segment by segment.Moreover, we adopt memory mechanism to preserve and update the history information dynamically, allowing the semantic flow across different windows.Experimental results on two large-scale datasets that consist of scientific papers demonstrate that our model substantially outperforms previous state-of-the-art models.Besides, we perform qualitative and quantitative investigations on how our model works and where the performance gain comes from. 1

Text summarization aims to compress a textual document to a short summary while keeping salient information.Extractive approaches are widely used in text summarization because of their fluency and efficiency.However, most of existing extractive models hardly capture intersentence relationships, particularly in long documents.They also often ignore the effect of topical information on capturing important contents.To address these issues, this paper proposes a graph neural network (GNN)-based extractive summarization model, enabling to capture intersentence relationships efficiently via graph-structured document representation.Moreover, our model integrates a joint neural topic model (NTM) to discover latent topics, which can provide document-level features for sentence selection.The experimental results demonstrate that our model not only substantially achieves state-of-the-art results on CNN/DM and NYT datasets but also considerably outperforms existing approaches on scientific paper datasets consisting of much longer documents, indicating its better robustness in document genres and lengths.Further discussions show that topical information can help the model preselect salient contents from an entire document, which interprets its effectiveness in long document summarization.