High-quality benchmarks are essential for evaluating reasoning and retrieval capabilities of large language models (LLMs). However, curating datasets for this purpose is not a permanent solution as they are prone to data leakage and inflated performance results. To address these challenges, we propose PhantomWiki: a pipeline to generate unique, factually consistent document corpora with diverse question-answer pairs. Unlike prior work, PhantomWiki is neither a fixed dataset, nor is it based on any existing data. Instead, a new PhantomWiki instance is generated on demand for each evaluation. We vary the question difficulty and corpus size to disentangle reasoning and retrieval capabilities, respectively, and find that PhantomWiki datasets are surprisingly challenging for frontier LLMs. Thus, we contribute a scalable and data leakage-resistant framework for disentangled evaluation of reasoning, retrieval, and tool-use abilities.

We present a communication scheme using guessing random additive noise decoding (GRAND) to improve flexibility and reliability of the existing compressed error (CE) framework. The CE framework uses information feedback to construct follow-up transmissions by compressing previous noise realizations, offering high reliability at a code rate close to the forward channel capacity. The channel decoding algorithm GRAND allows us to efficiently maintain this performance in noisy feedback settings by shifting redundancy for forward message protection to the feedback channel. Our scheme, GRAND-CE, is therefore appropriate for cases where forward and feedback channel usage costs are asymmetric, e.g. uplink communications. GRAND-CE offers super-exponential error rate performance as channel use increases, with finite usage of a noisy feedback channel. Unlike the traditional forward error correction model, the receiver performs error correction encoding and the sender handles decoding. We also propose a technique for pipelining sequential transmissions to maintain fixed forward transmission length and good feedback channel coding performance.