Language models are often miscalibrated, leading to confidently incorrect answers. We introduce GRACE, a benchmark for language model calibration that incorporates comparison with human calibration. GRACE consists of question-answer pairs, in which each question contains a series of clues that gradually become easier, all leading to the same answer; models must answer correctly as early as possible as the clues are revealed. This setting permits granular measurement of model calibration based on how early, accurately, and confidently a model answers. After collecting these questions, we host live human vs. model competitions to gather 1,749 data points on human and model teams’ timing, accuracy, and confidence. We propose a metric, CalScore, that uses GRACE to analyze model calibration errors and identify types of model miscalibration that differ from human behavior. We find that although humans are less accurate than models, humans are generally better calibrated. Since state-of-the-art models struggle on GRACE, it effectively evaluates progress on improving model calibration.

Nishant Balepur, Matthew Shu, Yoo Yeon Sung, Seraphina Goldfarb-Tarrant, Shi Feng, Fumeng Yang, Rachel Rudinger, Jordan Lee Boyd-Graber. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Yoo Yeon Sung, Maharshi Gor, Eve Fleisig, Ishani Mondal, Jordan Lee Boyd-Graber. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Training question answering (QA) and information retrieval systems for web queries require large, expensive datasets that are difficult to annotate and time-consuming to gather.Moreover, while natural datasets of informationseeking questions are often prone to ambiguity or ill-formed, there are troves of freely available, carefully crafted question datasets for many languages.Thus, we automatically generate shorter, information-seeking questions, resembling web queries in the style of the Natural Questions (NQ) dataset from longer trivia data.Training a QA system on these transformed questions is a viable strategy for alternating to more expensive training setups showing the F1 score difference of less than six points and contrasting the final systems. 1 We also combine NQ with QB-TRANS as training data in our supervised setting (Section 5), improving F1 (tested on NQ test set) by 10 points compared to training on only NQ.QB-TRANS lacks issues that plague NQ: presupposition and ambiguity (Section 6).Moreover, NATURALIZATION generalizes to other datasets (Section 6.4).Our contributions are naturalizing Manchester QB questions into Cranfield QB-TRANS while retaining the positive traits of QB samples, thereby improving QA with a more affordable process.The dataset generated from NATURALIZATION can be used to answer non-NQ data (Section 6.5) which proves the generalization of NATURALIZATION.Section 8 shows how this can ensure a cheaper and more up-to-date alternative to NQ data by generating large-scale Cranfield dataset that benefits training questionanswering models and generalizes to other non-NQ datasets. Artful but Arcane QB datasetThis section discusses why we use QB data and how different they are from NQ questions.The next section explains NATURALIZATION (Section 3).Elicitations from QB dataset Consider this QB example:

Polarization and the marketplace for impressions have conspired to make navigating information online difficult for users, and while there has been a significant effort to detect false or misleading text, multimodal datasets have received considerably less attention.To complement existing resources, we present multimodal Video Misleading Headline (VMH), a dataset that consists of videos and whether annotators believe the headline is representative of the video's contents.After collecting and annotating this dataset, we analyze multimodal baselines for detecting misleading headlines.Our annotation process also focuses on why annotators view a video as misleading, allowing us to better understand the interplay of annotators' background and the content of the videos.