Discrepancies between scientific papers and their code undermine reproducibility, a concern that grows as automated research agents scale scientific output beyond human review capacity.Whether LLMs can reliably detect such discrepancies has not been systematically measured.To this end, we present SCICOQA, a dataset of 635 paper-code discrepancies (92 real, 543 synthetic) for this cross-modal verification task.Across 22 evaluated models, even the best-performing LLMs, Gemini 3.1 Pro and GPT-5 Mini, detect only 46.7% of real-world discrepancies, revealing a critical gap in automated scientific quality assurance.We construct SCICOQA from GitHub issues and reproducibility papers, and propose a synthetic generation pipeline to scale beyond AI to Physics, Quantitative Biology, and other computational sciences.We further introduce a taxonomy of discrepancy types and categories to characterize the occurring mismatches.Our analysis shows that models particularly struggle with omitted paper details, long-context inputs, and papers outside their pre-training corpus.

Providing constructive feedback to paper authors is a core component of peer review.With reviewers increasingly having less time to perform reviews, automated support systems are required to ensure high reviewing quality, thus making the feedback in reviews useful for authors.To this end, we identify four key aspects of review comments (individual points in weakness sections of reviews) that drive the utility for authors: Actionability, Grounding & Specificity, Verifiability, and Helpfulness.To enable evaluation and development of models assessing review comments, we introduce the RevUtil dataset.We collect 1,430 human-labeled review comments and scale our data with 10k synthetically labeled comments for training purposes.The synthetic data additionally contains rationales, i.e., explanations for the aspect score of a review comment.Employing the RevUtil dataset, we benchmark fine-tuned models for assessing review comments on these aspects and generating rationales.Our experiments demonstrate that these fine-tuned models achieve agreement levels with humans comparable to, and in some cases exceeding, those of powerful closed models like GPT-4o.Our analysis further reveals that machinegenerated reviews generally underperform human reviews on our four aspects. 1

Tim Baumgärtner, Ted Briscoe, Iryna Gurevych. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Pairing a lexical retriever with a neural reranking model has set state-of-the-art performance on large-scale information retrieval datasets.This pipeline covers scenarios like question answering or navigational queries, however, for information-seeking scenarios, users often provide information on whether a document is relevant to their query in form of clicks or explicit feedback.Therefore, in this work, we explore how relevance feedback can be directly integrated into neural re-ranking models by adopting few-shot and parameterefficient learning techniques.Specifically, we introduce a kNN approach that re-ranks documents based on their similarity with the query and the documents the user considers relevant.Further, we explore Cross-Encoder models that we pre-train using meta-learning and subsequently fine-tune for each query, training only on the feedback documents.To evaluate our different integration strategies, we transform four existing information retrieval datasets into the relevance feedback scenario.Extensive experiments demonstrate that integrating relevance feedback directly in neural re-ranking models improves their performance, and fusing lexical ranking with our best performing neural reranker outperforms all other methods by 5.2% nDCG@20.

This paper introduces the PhotoBook dataset, a large-scale collection of visually-grounded, task-oriented dialogues in English designed to investigate shared dialogue history accumulating during conversation.Taking inspiration from seminal work on dialogue analysis, we propose a data-collection task formulated as a collaborative game prompting two online participants to refer to images utilising both their visual context as well as previously established referring expressions.We provide a detailed description of the task setup and a thorough analysis of the 2,500 dialogues collected.To further illustrate the novel features of the dataset, we propose a baseline model for reference resolution which uses a simple method to take into account shared information accumulated in a reference chain.Our results show that this information is particularly important to resolve later descriptions and underline the need to develop more sophisticated models of common ground in dialogue interaction. 1

Our goal is to explore how the abilities brought in by a dialogue manager can be included in end-to-end visually grounded conversational agents. We make initial steps towards this general goal by augmenting a task-oriented visual dialogue model with a decision-making component that decides whether to ask a follow-up question to identify a target referent in an image, or to stop the conversation to make a guess. Our analyses show that adding a decision making component produces dialogues that are less repetitive and that include fewer unnecessary questions, thus potentially leading to more efficient and less unnatural interactions.