Generation of citation-backed reports is a primary use case for retrieval-augmented generation (RAG) systems. While open-source evaluation tools exist for various RAG tasks, tools designed for report generation are lacking. Accordingly, we introduce Auto-ARGUE, a robust LLM-based implementation of the recently proposed ARGUE framework for report generation evaluation. We present analysis of Auto-ARGUE on the report generation pilot task from the TREC 2024 NeuCLIR track and on two tasks from the TREC 2024 RAG track, showing good system-level correlations with human judgments. Additionally, we release ARGUE-viz, a web app for visualization and fine-grained analysis of Auto-ARGUE judgments and scores1.

George Giannakopoulos, Jeff Kubina, John Conroy, Josef Steinberger, Benoit Favre, Mijail Kabadjov, Udo Kruschwitz, Massimo Poesio. Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2015.

An update summary should provide a fluent summarization of new information on a time-evolving topic, assuming that the reader has already reviewed older documents or summaries. In 2007 and 2008, an annual summarization evaluation included an update summarization task. Several participating systems produced update summaries indistinguishable from human-generated summaries when measured using ROUGE. However, no machine system performed near human-level performance in manual evaluations such as pyramid and overall responsiveness scoring. We present a metric called Nouveau-ROUGE that improves correlation with manual evaluation metrics and can be used to predict both the pyramid score and overall responsiveness for update summaries. Nouveau-ROUGE can serve as a less expensive surrogate for manual evaluations when comparing existing systems and when developing new ones.

Presents a hidden Markov model for automatic karyotyping. Previously, we demonstrated that this method is robust in the presence of different types of metaphase spreads, truncation of chromosomes and minor chromosome abnormalities, and that it gives results superior to neural networks on standard data sets. In this paper, we evaluate it on a data set consisting of a mix of chromosomes obtained from blood, amniotic fluid and bone marrow specimens. The method is shown to be robust on this mixed set of data, as well as giving far superior results than those obtained by neural networks.

A sentence extract summary of a document is a subset of the document's sentences that contains the main ideas in the document. We present an approach to generating such summaries, a hidden Markov model that judges the likelihood that each sentence should be contained in the summary. We compare the results of this method with summaries generated by humans, showing that we obtain significantly higher agreement than do earlier methods.

This paper analyzes two memory hot spot problems associated with massively parallel MIMD computers.The jirst is the memory stride problem, which is similar to stride problems found in existing supercomputers.Pseudo-random interleaving, as proposed by Norton and Melton, is the preferred solution.The second hot spot problem occurs in designs that use two separate memory accesses to lock and unlock critical sections (split transaction) and employ a jirst comeljirst serve queuing mechanism for shared memory locations.A bistabilip in throughput brought about by these conditions is analyzed and experimentally demonstrated.Simple equations are presented which predict the throughput at a critical section of code as a function of the number of applied threads.These equations also express the maximum number of threads that can safely be applied without the possibility of stalling.

As part of the Horizon project, a set of application programs is being written and their performance is being evaluated. Two of these applications, the fast Fourier transform and sparse matrix computations, are described. For both problems, efficient implementations are developed that take advantage of Horizon's unique features, including fined-grained synchronization, multiple instruction streams, and a horizontal instruction set. The results indicate that Horizon will offer very high performance, sometimes in excess of a sustained rate of one floating point operation per tick.>