David Wadden, Kejian Shi, Jacob Morrison, Alan Li, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, Doug Downey, Hannaneh Hajishirzi, Arman Cohan. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Scientific literature is typically dense, requiring significant background knowledge and deep comprehension for effective engagement.We introduce SCIDQA, a new dataset for reading comprehension that challenges LLMs for a deep understanding of scientific articles, consisting of 2,937 QA pairs.Unlike other scientific QA datasets, SCIDQA sources questions from peer reviews by domain experts and answers by paper authors, ensuring a thorough examination of the literature.We enhance the dataset's quality through a process that carefully filters out lower quality questions, decontextualizes the content, tracks the source document across different versions, and incorporates a bibliography for multi-document question-answering.Questions in SCIDQA necessitate reasoning across figures, tables, equations, appendices, and supplementary materials, and require multi-document reasoning.We evaluate several open-source and proprietary LLMs across various configurations to explore their capabilities in generating relevant and factual responses.Our comprehensive evaluation, based on metrics for surface-level similarity and LLM judgements, highlights notable performance discrepancies.SCIDQA represents a rigorously curated, naturally derived scientific QA dataset, designed to facilitate research on complex scientific text understanding.