Text classification is a well-established area of machine learning that involves automatically categorizing text into predefined categories, such as positive or negative in sentiment analysis. It typically involves applying a machine learning algorithm to learn a predictive model from a set of labeled training texts and using the model to classify new texts. Large language models (LLMs) have been successfully applied to various natural language processing tasks, including text classification. There are two main approaches to text classification with LLMs: In-Context Learning and Fine-Tuning. In-Context Learning involves prompt engineering, allowing the model to learn a new task using only a few demonstration examples, while Fine-Tuning adjusts the model’s parameters with additional labeled data. Retrieval-Augmented Generation (RAG) is a retrieval process that enhances LLM performance by selecting relevant examples. This paper presents our work on utilizing In-Context Learning and RAG to identify delay-related statements during the document review process of construction delay disputes. We also report the results of our experiments comparing the accuracy of In-Context Learning with that of traditional machine learning algorithms, such as logistic regression and KNN.

This paper presents a comparative study evaluating the performance of Large Language Models (LLMs) in generating timeline summaries from construction delay documents. We assessed seven open-source LLMs and two commercial chatbots (ChatGPT and Claude) on their ability to extract, organize, and summarize delay events from twenty-one carefully curated synthetic snippets of text. The evaluation framework combined automatic metrics (BERTScore and ROUGE scores) with expert human assessment across four dimensions: event description accuracy, date accuracy, event capture completeness, and language quality.Results demonstrate that while commercial solutions, particularly Claude, achieved superior performance, several open-source alternatives showed comparable capabilities. Notably, Llama-3.1-70B-Instruct showed robust performance in event capture and source tracking, while Llama-3.1-8B-Instruct offered efficient processing with balanced performance among smaller models. A critical finding was the widespread challenge in temporal information processing, with only Claude achieving complete accuracy in date extraction and event association. The study's findings suggest that open-source LLMs can serve as practical tools for construction document analysis, although model selection is a critical consideration based on specific accuracy and efficiency requirements, and resource constraints.

In the realm of U.S. legal practice, safeguarding privileged communications and data from unintended disclosure is of utmost importance. Traditionally, attorneys have relied on keyword searches and manual reviews to find privileged documents in legal cases. However, as data volumes grow, this method becomes increasingly costly and inefficient. Machine learning techniques have emerged as a viable solution for identifying privileged documents. This paper introduces a hybrid approach to identifying privileged documents by integrating rule-based modeling logistic regression, and active learning within a predictive analytics framework. This method streamlines privilege prediction in legal datasets by leveraging metadata, such as attorney and firm details, along with document text, to create a comprehensive end-to-end process for privilege identification. Evaluation through precision and recall metrics on the dataset from a real-world legal matter demonstrates the effectiveness of this approach in classifying privileged status in complex legal documents, providing a scalable and accurate solution for automated document review in legal contexts.

Corporate litigation costs associated with manually reviewing documents in legal matters continue to rise, driven by the ever-expanding volumes of data generated by businesses. To address this challenge, legal teams have increasingly turned to text classification techniques, commonly referred to in the legal community as predictive modeling, predictive coding, or "Technology Assisted Review (TAR)", to identify relevant documents more efficiently during the discovery phase of legal matters. By streamlining the classification process, predictive coding has proven instrumental in minimizing both the time and cost associated with legal discovery. As more corporations expand their operations across global regions, legal matters increasingly involve multilingual datasets, introducing significant complexities for the application of predictive coding. Traditional predictive models exhibit bias towards the dominant language within the model’s training data. The prevalence of multiple languages within the same document further complicates the challenge. This can lead to inefficiencies, elevated costs, and risks of inaccurate classifications of nondominant language documents.This research empirically examines two methods for applying predictive coding to multilingual datasets: a single multilingual model approach and a language-specific modeling approach. This group of collaborators posits that language-specific models will outperform the single multilanguage model approach in evaluating document relevance. The research used a multilingual dataset from a recent confidential real-world legal matter containing predominantly English and Chinese document text Performance was assessed using precision and recall measures. Our results show that the ensemble approach – employing language-specific models for classifying documents in their respective languages and employing the mixed language model across all remaining documents—outperforms a single multilanguage model. The outcomes of this research aim to inform the workflow considerations for legal professionals undertaking multilingual discovery efforts. The findings will help legal practitioners make data-driven decisions on the most efficient and accurate modeling approach for multilingual document review, shaping the future of predictive coding in cross-linguistic e-discovery.

Email threading is a commonly used tool by legal practitioners to streamline document review and classification in legal proceedings. Threading organizes component pieces of an email thread together to effectively reduce a dataset. The most inclusive threads and their associated document attachments are maintained, and non-inclusive or duplicative thread components are set aside. However, as data volumes continue to grow and outpace deadlines for legal proceedings, practitioners often look to incorporate multiple cost-effective and defensible technology solutions to further reduce or otherwise accelerate document review and classification. One such technology is predictive modeling – known in the legal industry as ‘predictive coding’ or ‘Technology Assisted Review (TAR)’ – which is a popular tool used to augment a manual document review and classification process. Like email threading, predictive coding has become more commonplace recently for its proven ability to minimize manual document classification, thus reducing the time and cost associated with this aspect of legal proceedings.In this study, we explore the performance impacts of layering predictive modeling onto an email threading reduction workflow. Generally, a predictive model is established first, and email threading is layered onto the scored output to further reduce and streamline document review. Our research evaluates a reversed workflow, where a population is initially reduced to its inclusive email threads, and this limited dataset is used to train and apply a predictive model to the larger population of documents for classification. Using classified data from four real-world legal proceedings, we compare the performance impact of email threading on predictive modeling by 1) training a model using all positive and negative examples, 2) training a model using positive and negative examples from only inclusive email threads, and 3) training a model using positive examples from only inclusive email threads, but negative examples from all emails. The results of our research provide thoughtful, empirical insights for legal practitioners to review when exploring the deployment of both email threading and predictive modeling into a single, cohesive document classification strategy for their legal proceedings.

The costs involved in manually reviewing documents in legal civil litigations have grown dramatically as more and more information is stored electronically. As a result, the document review process can require an extraordinary dedication of resources. In construction litigations, quickly finding supporting documentation in a delay dispute is critical to the success of a matter. Identifying relevant delay-related communications and supporting documentation has historically been expensive and time consuming. Using machine learning technologies, respondents can be more comprehensive in their assessment of the data requiring review to respond to the claim in time. Explainable machine learning is an active machine learning research area, and in an explainable machine learning system, predictions generated from a machine learning model are explainable and human understandable. In delay dispute ‘document review’ scenarios, a document can be identified as delay-related, as long as one or more of the text snippets in a document are deemed delay-related. In these scenarios, if these delay-related snippets can be located, then attorneys could easily evaluate the model’s decision. The authors of this paper propose an approach for accurately identifying rationales and an approach for boosting document classification accuracy using delay-related snippets and their applications in construction delay disputes. The authors conducted experiments using data from a few real world delay dispute matters and the results from these experiments show that the proposed approaches have the potential to significantly advance the application of text classification in document review in construction delay dispute matters.

Predictive modeling - known in the legal industry as ‘predictive coding’ or ‘Technology Assisted Review (TAR)’ - is a popular tool used by legal professionals to augment a historically manual document review and classification process in responding to data requests for various legal proceedings. It has become more commonplace over the past decade due to its proven ability to minimize manual document classification, thus reducing the time and cost associated with this aspect of legal proceedings. There is significant research supporting the effectiveness of this technology that includes topics, such as identifying the most performant machine learning algorithms (e.g., logistic regression) or establishing the best methodologies for selecting representative training and testing data. Primarily, this research has been performed without a focus on multilanguage data sets because many legal proceedings typically involve one primary, dominant language. As acceptance of predictive modeling technology grows, legal practitioners have developed independent preferences for handling multiple languages in data sets. These preferences have been primarily based on anecdotal experience rather than empirical assessments and have resulted in two modeling approaches. The first approach uses a single model, which is less complex, and more cost effective than the next approach. The second develops multiple language-specific models, which can create workflow complexity that may impact the overall cost savings that predictive coding seeks to achieve. Proponents of the second approach believe that creating models per language results in better performance over a single model approach. In this study, we empirically explore the performance differences between the two approaches - single model and multiple language-specific models. We hypothesize that a single model approach performs similarly to a language-specific modeling approach when classifying documents for relevance. We evaluate both approaches by comparing their resulting precision-recall curves across real-world test data from legal document review projects. Excitingly, our results demonstrate that in most scenarios, a single model (trained with multiple languages) approach will perform as well, or sometimes better than the approach that uses a group of language-specific models. The results of our research provide a roadmap for legal practitioners and other parties to thoughtfully engage in dialogue around the most effective way to deploy predictive coding within a multilanguage data set. This research will change how predictive models are deployed for multilanguage data sets across the legal industry and enable significant cost savings for clients and legal practitioners by utilizing the most efficient modeling method.

The increased integration of Large Language Models (LLMs) across industry sectors is enabling domain experts with new text classification optimization methods. These LLMs are pretrained on exceedingly large amounts of data; however, practitioners can perform additional training, or “fine-tuning,” to improve their text classifier’s results for their own use cases. This paper presents a series of experiments comparing a standard, pretrained DistilBERT model and a fine-tuned DistilBERT model, both leveraged for the downstream NLP task of text classification. Tuning the model using domain-specific data from real-world legal matters suggests fine-tuning improves the performance of LLM text classifiers.To evaluate the performance of text classification models, using these two Large Language Models, we employed two distinct approaches that 1) score a whole document’s text for prediction and 2) score snippets (sentence-level components of a document) of text for prediction. When comparing the two approaches we found that one prediction method outperforms the other, depending on the use case.

In the domain of natural language processing and text analysis, the Bidirectional Encoder Representations from Transformers (BERT) has emerged as a powerful tool for discerning the intricate nuances of textual data. Nonetheless, BERT’s inherent token limit of 512 tokens presents a noteworthy challenge when confronted with exceedingly long documents. Lengthy documents are commonly encountered during legal document review and often exceed the 512-token constraint. In response to this challenge, this study empirically compares two distinct applications of BERT, leveraging real-world, construction industry legal data. The approach compares applications of BERT to the entire document and on segmented text portions from each document. In the latter approach, the highest-scoring text segment from each document represents the document’s score. This research offers practical insights for effectively utilizing BERT in scenarios where document length exceeds the token limit. Our results allow practitioners and researchers to make informed choices when confronted with documents of significant length, thus contributing to a more effective and insightful application of BERT for text analysis.

Protecting privileged communications and data from disclosure is paramount for legal teams. Unrestricted legal advice, such as attorney-client communication or litigation strategy, is exempt from disclosure in litigation or regulatory events and is vital to the attorney-client relationship. To protect this information from disclosure, companies and outside counsel must review vast amounts of documents to determine those that contain privileged material. This process is costly and time consuming. As data volumes increase, legal counsel employs methods to reduce the number of documents requiring review while balancing the need to ensure the protection of privileged information. Keyword searching is a popular method to target privileged information and reduce document review populations. Keyword terms are effective at casting a wide net but generally return overly inclusive results – most of which do not contain privileged information. To overcome the weaknesses of keyword searching, legal teams have started using supervised learning techniques to more precisely target privileged information. However, reviewing and labeling training documents is costly and time intensive and may cause counsel to forego the use of supervised learning in certain scenarios. In addition, supervised learning techniques may not find all the privileged documents in a document review and require companies to use keyword terms to identify critical privileged information. In this paper, the authors propose a novel method to automatically identify privileged documents without the need to label new training documents. This method integrates rule-based reasoning with transfer learning. Experimental results show that the proposed integrated method performs better than rule-based reasoning and transfer learning individually and can effectively identify privileged documents.

US corporations regularly spend millions of dollars reviewing electronically-stored documents in legal matters. Recently, attorneys apply text classification to efficiently cull massive volumes of data to identify responsive documents for use in these matters. While text classification is regularly used to reduce the discovery costs of legal matters, it also faces a perception challenge: amongst lawyers, this technology is sometimes looked upon as a "black box." Put simply, no extra information is provided for attorneys to understand why documents are classified as responsive. In recent years, explainable machine learning has emerged as an active research area. In an explainable machine learning system, predictions or decisions made by a machine learning model are human understandable. In legal ‘document review’ scenarios, a document is responsive, because one or more of its small text snippets are deemed responsive. In these scenarios, if these responsive snippets can be located, then attorneys could easily evaluate the model’s document classification decisions – this is especially important in the field of responsible AI. Our prior research identified that predictive models created using annotated training text snippets improved the precision of a model when compared to a model created using all of a set of documents’ text as training. While interesting, manually annotating training text snippets is not generally practical during a legal document review. However, small increases in precision can drastically decrease the cost of large document reviews. Automating the identification of training text snippets without human review could then make the application of training text snippet-based models a practical approach. This paper proposes two simple machine learning methods to locate responsive text snippets within responsive documents without using human annotated training text snippets. The two methods were evaluated and compared with a document classification method using three datasets from actual legal matters. The results show that the two proposed methods outperform the document-level training classification method in identifying responsive text snippets in responsive documents. Additionally, the results suggest that we can automate the successful identification of training text snippets to improve the precision of our predictive models in legal document review and thereby help reduce the overall cost of review.

Protecting privileged communications and data from disclosure is paramount for legal teams. Legal advice, such as attorney-client communications or litigation strategy are typically exempt from disclosure in litigations or regulatory events and are vital to the attorney-client relationship. To protect this information from disclosure, companies and outside counsel often review vast amounts of documents to determine those that contain privileged material. This process is extremely costly and time consuming. As data volumes increase, legal counsel normally employs methods to reduce the number of documents requiring review while balancing the need to ensure the protection of privileged information. Keyword searching is relied upon as a method to target privileged information and reduce document review populations. Keyword searches are effective at casting a wide net but often return overly inclusive results – most of which do not contain privileged information. To overcome the weaknesses of keyword searching, legal teams increasingly are using machine learning techniques to target privileged information. In these studies, classic text classification techniques are applied to build classification models to identify privileged documents. In this paper, the authors propose a different method by applying machine learning / convolutional neural network techniques (CNN) to identify privileged documents. Our proposed method combines keyword searching with CNN. For each keyword term, a CNN model is created using the context of the occurrences of the keyword. In addition, a method was proposed to select reliable privileged (positive) training keyword occurrences from labeled positive training documents. Extensive experiments were conducted, and the results show that the proposed methods can significantly reduce false positives while still capturing most of the true positives.

Though technology assisted review in electronic discovery has been focusing on text data, the need of advanced analytics to facilitate reviewing multimedia content is on the rise. In this paper, we present several applications of deep learning in computer vision to Technology Assisted Review of image data in legal industry. These applications include image classification, image clustering, and object detection. We use transfer learning techniques to leverage established pretrained models for feature extraction and fine tuning. These applications are first of their kind in the legal industry for image document review. We demonstrate effectiveness of these applications with solving real world business challenges.

Research has shown that Convolutional Neural Networks (CNN) can be effectively applied to text classification as part of a predictive coding protocol. That said, most research to date has been conducted on data sets with short documents that do not reflect the variety of documents in real world document reviews. Using data from four actual reviews with documents of varying lengths, we compared CNN with other popular machine learning algorithms for text classification, including Logistic Regression, Support Vector Machine, and Random Forest. For each data set, classification models were trained with different training sample sizes using different learning algorithms. These models were then evaluated using a large randomly sampled test set of documents, and the results were compared using precision and recall curves. Our study demonstrates that CNN performed well, but that there was no single algorithm that performed the best across the combination of data sets and training sample sizes. These results will help advance research into the legal profession's use of machine learning algorithms that maximize performance.

Companies regularly spend millions of dollars producing electronically-stored documents in legal matters. Over the past two decades, attorneys have been using a variety of technologies to conduct this exercise, and most recently, parties on both sides of the `legal aisle' are accepting the use of machine learning techniques like text classification to cull massive volumes of data and to identify responsive documents for use in these matters. While text classification is regularly used to reduce the discovery costs in legal matters, text classification also faces a peculiar perception challenge: amongst lawyers, this technology is sometimes looked upon as a black box Put simply, very little information is provided for attorneys to understand why documents are classified as responsive. In recent years, a group of AI and Machine Learning researchers have been actively researching Explainable AI. In an explainable AI system, actions or decisions are human understandable. In legal `document review' scenarios, a document can be identified as responsive, as long as one or more of the text snippets (small passages of text) in a document are deemed responsive. In these scenarios, if text classification can be used to locate these responsive snippets, then attorneys could easily evaluate the model's document classification decision. When deployed with defined and explainable results, text classification can drastically enhance the overall quality and speed of the document review process by reducing the time it takes to review documents. Moreover, explainable predictive coding provides lawyers with greater confidence in the results of that supervised learning task. This paper describes a framework for explainable text classification as a valuable tool in legal services: for enhancing the quality and efficiency of legal document review and for assisting in locating responsive snippets within responsive documents. This framework has been implemented in our legal analytics product, which has been used in hundreds of legal matters. We also report our experimental results using the data from an actual legal matter that used this type of document review.

In today's legal environment, lawsuits and regulatory investigations require companies to embark upon increasingly intensive data-focused engagements to identify, collect and analyze large quantities of data. When documents are staged for review - where they are typically assessed for relevancy or privilege - the process can require companies to dedicate an extraordinary level of resources, both with respect to human resources, but also with respect to the use of technology-based techniques to intelligently sift through data. Companies regularly spend millions of dollars producing `responsive' electronically-stored documents for these types of matters. For several years, attorneys have been using a variety of tools to conduct this exercise, and most recently, they are accepting the use of machine learning techniques like text classification (referred to as predictive coding in the legal industry) to efficiently cull massive volumes of data to identify responsive documents for use in these matters. In recent years, a group of AI and Machine Learning researchers have been actively researching Explainable AI. In an explainable AI system, actions or decisions are human understandable. In typical legal `document review' scenarios, a document can be identified as responsive, as long as one or more of the text snippets (small passages of text) in a document are deemed responsive. In these scenarios, if predictive coding can be used to locate these responsive snippets, then attorneys could easily evaluate the model's document classification decision. When deployed with defined and explainable results, predictive coding can drastically enhance the overall quality and speed of the document review process by reducing the time it takes to review documents. Moreover, explainable predictive coding provides lawyers with greater confidence in the results of that supervised learning task. The authors of this paper propose the concept of explainable predictive coding and simple explainable predictive coding methods to locate responsive snippets within responsive documents. We also report our preliminary experimental results using the data from an actual legal matter that entailed this type of document review. The purpose of this paper is to demonstrate the feasibility of explainable predictive coding in the context of professional services in the legal space.

Protecting privileged communications and data from disclosure is paramount for legal teams. Unrestricted legal advice, such as attorney-client communications or litigation strategy. are vital to the legal process and are exempt from disclosure in litigations or regulatory events. To protect this information from being disclosed, companies and outside counsel must review vast amounts of documents to determine those that contain privileged material. This process is extremely costly and time consuming. As data volumes increase, legal counsel employ methods to reduce the number of documents requiring review while balancing the need to ensure the protection of privileged information. Keyword searching is relied upon as a method to target privileged information and reduce document review populations. Keyword searches are effective at casting a wide net but return over inclusive results - most of which do not contain privileged information - and without detailed knowledge of the data, keyword lists cannot be crafted to find all privilege material. Overly-inclusive keyword searching can also be problematic, because even while it drives up costs, it also can cast `too far of a net' and thus produce unreliable results.To overcome these weaknesses of keyword searching, legal teams are using a new method to target privileged information called predictive modeling. Predictive modeling can successfully identify privileged material but little research has been published to confirm its effectiveness when compared to keyword searching. This paper summarizes a study of the effectiveness of keyword searching and predictive modeling when applied to real- world data. With this study, this group of collaborators wanted to examine and understand the benefits and weaknesses of both approaches to legal teams with identifying privilege material in document populations.

Training documents have a significant impact on the performance of predictive models in the legal domain. Yet, there is limited research that explores the effectiveness of the training document selection strategy - in particular, the strategy used to select the seed set, or the set of documents an attorney reviews first to establish an initial model. Since there is limited research on this important component of predictive coding, the authors of this paper set out to identify strategies that consistently perform well. Our research demonstrated that the seed set selection strategy can have a significant impact on the precision of a predictive model. Enabling attorneys with the results of this study will allow them to initiate the most effective predictive modeling process to comb through the terabytes of data typically present in modern litigation. This study used documents from four actual legal cases to evaluate eight different seed set selection strategies. Attorneys can use the results contained within this paper to enhance their approach to predictive coding.

One type of machine learning, text classification, is now regularly applied in the legal matters involving voluminous document populations because it can reduce the time and expense associated with the review of those documents. One form of machine learning - Active Learning - has drawn attention from the legal community because it offers the potential to make the machine learning process even more effective. Active Learning, applied to legal documents, is considered a new technology in the legal domain and is continuously applied to all documents in a legal matter until an insignificant number of relevant documents are left for review. This implementation is slightly different than traditional implementations of Active Learning where the process stops once achieving acceptable model performance. The purpose of this paper is twofold: (i) to question whether Active Learning actually is a superior learning methodology and (ii) to highlight the ways that Active Learning can be most effectively applied to real legal industry data. Unlike other studies, our experiments were performed against large data sets taken from recent, real-world legal matters covering a variety of areas. We conclude that, although these experiments show the Active Learning strategy popularly used in legal document review can quickly identify informative training documents, it becomes less effective over time. In particular, our findings suggest this most popular form of Active Learning in the legal arena, where the highest-scoring documents are selected as training examples, is in fact not the most efficient approach in most instances. Ultimately, a different Active Learning strategy may be best suited to initiate the predictive modeling process but not to continue through the entire document review.

Predictive coding, once used in only a small fraction of legal and business matters, is now widely deployed to quickly cull through increasingly vast amounts of data and reduce the need for costly and inefficient human document review. Previously, the sole front-end input used to create a predictive model was the exemplar documents (training data) chosen by subject-matter experts. Many predictive coding tools require users to rely on static preprocessing parameters and a single machine learning algorithm to develop the predictive model. Little research has been published discussing the impact preprocessing parameters and learning algorithms have on the effectiveness of the technology. A deeper dive into the generation of a predictive model shows that the settings and algorithm can have a strong effect on the accuracy and efficacy of a predictive coding tool. Understanding how these input parameters affect the output will empower legal teams with the information they need to implement predictive coding as efficiently and effectively as possible. This paper outlines different preprocessing parameters and algorithms as applied to multiple real-world data sets to understand the influence of various approaches.