Discovery is an important aspect of the civil litigation process in the United States of America, in which all parties to a lawsuit are permitted to request relevant evidence from other parties. With the rapid growth of digital content, the emerging need for “e-discovery” has created a strong demand for techniques that can be used to review massive collections both for “responsiveness” (i.e., relevance) to the request and for “privilege” (i.e., presence of legally protected content that the party performing the review may have a right to withhold). In this process, the party performing the review may incur costs of two types, namely, annotation costs (deriving from the fact that human reviewers need to be paid for their work) and misclassification costs (deriving from the fact that failing to correctly determine the responsiveness or privilege of a document may adversely affect the interests of the parties in various ways). Relying exclusively on automatic classification would minimize annotation costs but could result in substantial misclassification costs, while relying exclusively on manual classification could generate the opposite consequences. This article proposes a risk minimization framework (called MINECORE, for “minimizing the expected costs of review”) that seeks to strike an optimal balance between these two extreme stands. In MINECORE (a) the documents are first automatically classified for both responsiveness and privilege, and then (b) some of the automatically classified documents are annotated by human reviewers for responsiveness (typically by junior reviewers) and/or, in cascade, for privilege (typically by senior reviewers), with the overall goal of minimizing the expected cost (i.e., the risk ) of the entire process. Risk minimization is achieved by optimizing, for both responsiveness and privilege, the choice of which documents to manually review. We present a simulation study in which classes from a standard text classification test collection (RCV1-v2) are used as surrogates for responsiveness and privilege. The results indicate that MINECORE can yield substantially lower total cost than any of a set of strong baselines.

When searching for evidence in civil litigation, parties to a lawsuit have the right to withhold some content on grounds of specific privileges that serve to foster socially desirable outcomes such as open communication between attorneys and their clients. As inadvertent disclosure of privileged content can adversely impact a client's interests, review for privilege is a high-stakes process that is most often performed manually. Because the circumstances in which privilege can be claimed are generally well defined, review for privilege is amenable to some degree of automation. This paper describes the design of an interactive system to support privilege review in which the goals are to improve the speed and accuracy of privilege review. Results are reported for a within-subjects study in which six reviewers with different levels of expertise examined email for attorney-client privilege or any other valid basis for withholding the content from release. Quantitative results indicate that substantial and statistically significant improvements in recall can be achieved, but no significant differences in average review speed were detected. Participants self-reported that the identity features exposed by the system were most useful to them, and that the present implementation of features based on content or date added no discernible additional value.

In some jurisdictions, parties to a lawsuit can request documents from each other, but documents subject to a claim of privilege may be withheld. The TREC 2010 Legal Track developed what is presently the only public test collection for evaluating privilege classification. This paper examines the reliability and reusability of that collection. For reliability, the key question is the extent to which privilege judgments correctly reflect the opinion of the senior litigator whose judgment is authoritative. For reusability, the key question is the degree to which systems whose results contributed to creation of the test collection can be fairly compared with other systems that use those privilege judgments in the future. These correspond to measurement error and sampling error, respectively. The results indicate that measurement error is the larger problem.