Review for responsiveness is a recall-oriented document classification task central to civil litigation. In large legal matters, it may involve the coding of millions of documents by teams of dozens to hundreds of contract attorneys. We describe a prototype document review system based on a large language model (LLM) for replacing the first level of attorney review. Our system accepts the same guidance—a written review protocol—that would be provided to a human review team. We tested our prototype in the context of a live legal matter, evaluating both human review and our LLM-based system against a gold standard coded by expert senior attorneys. Our prototype achieved an estimated 96% recall and 60% precision without matter-specific tuning, and has numerous avenues for further improvement.

High Recall Retrieval (HRR) tasks, including eDiscovery in the law, systematic literature reviews, and sunshine law requests focus on efficiently prioritizing relevant documents for human review.Technology-assisted review (TAR) refers to iterative human-in-the-loop workflows that combine human review with IR and AI techniques to minimize both time and manual effort while maximizing recall. This full-day tutorial provides a comprehensive introduction to TAR. The morning session presents an overview of the key technologies and workflow designs used, the basics of practical evaluation methods, and the social and ethical implications of TAR deployment. The afternoon session provides more technical depth on the implications of TAR workflows for supervised learning algorithm design, how generative AI is can be applied in TAR, more sophisticated statistical evaluation techniques, and a wide range of open research questions.

Technology-assisted review (TAR) workflows are central to electronic discovery (eDiscovery). Researchers have proposed many methods for evaluating TAR workflows, but this research has had little impact on eDiscovery practice. We examine the operational constraints faced by eDiscovery reviewers and managers, and show how past evaluation proposals are inconsistent with their needs. We then present a new evaluation approach for one-phase TAR workflows based on confidence sequences. Our approach provides a review manager with complete control over the design and duration of the TAR workflow, as well as the amount and timing of review of evaluation documents. Evaluation documents can be reused for supervised learning while preserving valid frequentist confidence intervals on recall at all points during review. The method is expensive in terms of sample size but plausible for large scale reviews, and has many opportunities for improvement.