Benchmark collections have long enabled controlled comparison and cumulative progress in Information Retrieval (IR). However, prior meta-analyses show that reported effectiveness gains often fail to accumulate, in part due to weak or outdated baselines. Large language models (LLMs) are increasingly used in retrieval pipelines, yet their impact on established IR benchmarks has not been systematically analyzed. We analyze 179 publications reporting on the TREC Robust04 collection and the TREC Deep Learning 2020 (DL20) Passage Retrieval benchmark, using ACM Digital Library keyword search supplemented by citation-graph backtracking for Robust04. We observe what we term an LLM effect: recent systems incorporating LLM components achieve 8.8% higher nDCG@10 on DL20 than the best TREC 2020 result and 11.9% higher on Robust04 than the strongest pre-2024 result. However, evaluation practice has shifted from MAP to nDCG@10 over the same window, and our adaptation of the Data Contamination Quiz reveals 12-41% contamination across two widely-used LLM rerankers. Filtering contaminated topics shows no statistically significant effectiveness difference, but small samples and the uncertainty of adapting contamination detection to reranking prevent us from ruling out memorization as a contributing factor. We read the LLM effect as real but unverified: visible in the aggregate numbers, but not cleanly separable from metric drift or pretraining overlap.

Privacy-preserving data analysis must carefully balance the need for secure, meaningful computation on sensitive personal data with the fundamental rights of individuals to retain control over their information. Solid (Social Linked Data) presents an open protocol where users store and manage their data in personal, access-controlled pods. However, its potential for integration as a decentralized data store into existing infrastructures for privacy-preserving computations remains underexplored. We address how Solid can be effectively integrated into such platforms to support decentralized data sharing while meeting the technical requirements of privacy-aware research. To address this, we propose the Solid Gateway , a mediator that facilitates consent-driven access to Solid Pods within existing analysis environments. The Solid Gateway introduces request-specific authentication and authorization, manages access permissions, and orchestrates the retrieval of only the data necessary to fulfill individual data requests. Central to this approach is a novel granular data-sharing strategy, which restructures user data into minimal request-specific subsets, thus reducing unnecessary data transfers and limiting the exposure of irrelevant information. This ensures that contributors retain sovereignty over their data, while allowing privacy-preserving analysis to operate on decentralized sources. Our experimental evaluation, conducted on controlled artificial datasets, confirms the feasibility of our integration. The results demonstrate a significant reduction in data exposure while achieving improved data retrieval performance compared to existing approaches. Also, we compare our proposed solution against the WellFort architecture and demonstrate that our approach offers competitive fetch performance and significantly improves processing efficiency. Although the controlled nature of the evaluation limits comparability with existing platforms, it provides a reproducible foundation for future studies and practical deployments. This work contributes a concrete, extensible design for combining Solid with privacy-preserving computation, identifies key tradeoffs between privacy, performance, and system complexity, and opens pathways for future research into SPARQL integration, validation with established datasets, and the application of FAIR principles within Solid .

Navigating the vast and rapidly increasing sea of academic publications to identify institutional synergies, benchmark research contributions and pinpoint key research contributions has become an increasingly daunting task, especially with the current exponential increase in new publications. Existing tools provide useful overviews or single-document insights, but none supports structured, qualitative comparisons across institutions or publications. To address this, we demonstrate Compare, a novel framework that tackles this challenge by enabling sophisticated long-context comparisons of scientific contributions. Compare empowers users to explore and analyze research overlaps and differences at both the institutional and publication granularity, all driven by user-defined questions and automatic retrieval over online resources. For this we leverage on Retrieval-Augmented Generation over evolving data sources to foster long context knowledge synthesis. Unlike traditional scientometric tools, Compare goes beyond quantitative indicators by providing qualitative, citation-supported comparisons.

This paper analyzes the most common problems limiting reproducibility of Information Retrieval research and provides researchers with insights and guidelines to improve the reproducibility of experiments and to allow the verification of obtained results. We conducted a study on 45 reproduction reports off 17 different papers, which have been published at renowned IR conferences. We analyzed the reports qualitatively and quantitatively and looked into the different insights from different groups. Occurring problems are classified into three problem families and 13 categories and afre then analyzed with respect to their influence on the reproduction process as well as on their frequency of appearance over time and per conference. Of these 17 different papers, 14 papers were reproducible to a certain degree without significant differences to the original results, but in many cases not the whole experiment was reproducible due to missing code, information or data. Also, we look at assumptions that were made when reproducing the different papers, as some experiment workflows were incomplete and information was missing. In addition, we propose recommendations to make machine learning research more reproducible and FAIR.

Data in today's dynamic world undergoes constant change and evolution, spanning various formats such as text, websites, tweets, and sensor readings. Storing and referencing these diverse data types pose significant challenges due to data movement, changes in content or structure, and limited availability. Efficient data identification is crucial for speeding up scientific discovery and result validation, especially when data accessibility is guaranteed. Recent years have witnessed progress in data citation practices, with conferences mandating the inclusion of utilized and generated data. However, existing solutions primarily cater to static datasets, rendering them ineffective for dynamically evolving ones. This paper addresses this gap by providing a tailored dynamic data citation prototype for the International Soil Moisture Network, one of the largest scientific in situ soil moisture databases. Our work encompasses the implementation and evaluation of different data versioning strategies and a query store architecture that enables the citation, reproducibility, and verification of large sets of SQL queries to recreate data requests by users. By applying the RDA Dynamic Data Citation Guidelines, we assess the necessary needs for such a system and further measure the performance and storage impact of our proposed approaches.