This paper presents a comparative study evaluating the performance of Large Language Models (LLMs) in generating timeline summaries from construction delay documents. We assessed seven open-source LLMs and two commercial chatbots (ChatGPT and Claude) on their ability to extract, organize, and summarize delay events from twenty-one carefully curated synthetic snippets of text. The evaluation framework combined automatic metrics (BERTScore and ROUGE scores) with expert human assessment across four dimensions: event description accuracy, date accuracy, event capture completeness, and language quality.Results demonstrate that while commercial solutions, particularly Claude, achieved superior performance, several open-source alternatives showed comparable capabilities. Notably, Llama-3.1-70B-Instruct showed robust performance in event capture and source tracking, while Llama-3.1-8B-Instruct offered efficient processing with balanced performance among smaller models. A critical finding was the widespread challenge in temporal information processing, with only Claude achieving complete accuracy in date extraction and event association. The study's findings suggest that open-source LLMs can serve as practical tools for construction document analysis, although model selection is a critical consideration based on specific accuracy and efficiency requirements, and resource constraints.

In the realm of U.S. legal practice, safeguarding privileged communications and data from unintended disclosure is of utmost importance. Traditionally, attorneys have relied on keyword searches and manual reviews to find privileged documents in legal cases. However, as data volumes grow, this method becomes increasingly costly and inefficient. Machine learning techniques have emerged as a viable solution for identifying privileged documents. This paper introduces a hybrid approach to identifying privileged documents by integrating rule-based modeling logistic regression, and active learning within a predictive analytics framework. This method streamlines privilege prediction in legal datasets by leveraging metadata, such as attorney and firm details, along with document text, to create a comprehensive end-to-end process for privilege identification. Evaluation through precision and recall metrics on the dataset from a real-world legal matter demonstrates the effectiveness of this approach in classifying privileged status in complex legal documents, providing a scalable and accurate solution for automated document review in legal contexts.

Corporate litigation costs associated with manually reviewing documents in legal matters continue to rise, driven by the ever-expanding volumes of data generated by businesses. To address this challenge, legal teams have increasingly turned to text classification techniques, commonly referred to in the legal community as predictive modeling, predictive coding, or "Technology Assisted Review (TAR)", to identify relevant documents more efficiently during the discovery phase of legal matters. By streamlining the classification process, predictive coding has proven instrumental in minimizing both the time and cost associated with legal discovery. As more corporations expand their operations across global regions, legal matters increasingly involve multilingual datasets, introducing significant complexities for the application of predictive coding. Traditional predictive models exhibit bias towards the dominant language within the model’s training data. The prevalence of multiple languages within the same document further complicates the challenge. This can lead to inefficiencies, elevated costs, and risks of inaccurate classifications of nondominant language documents.This research empirically examines two methods for applying predictive coding to multilingual datasets: a single multilingual model approach and a language-specific modeling approach. This group of collaborators posits that language-specific models will outperform the single multilanguage model approach in evaluating document relevance. The research used a multilingual dataset from a recent confidential real-world legal matter containing predominantly English and Chinese document text Performance was assessed using precision and recall measures. Our results show that the ensemble approach – employing language-specific models for classifying documents in their respective languages and employing the mixed language model across all remaining documents—outperforms a single multilanguage model. The outcomes of this research aim to inform the workflow considerations for legal professionals undertaking multilingual discovery efforts. The findings will help legal practitioners make data-driven decisions on the most efficient and accurate modeling approach for multilingual document review, shaping the future of predictive coding in cross-linguistic e-discovery.

Email threading is a commonly used tool by legal practitioners to streamline document review and classification in legal proceedings. Threading organizes component pieces of an email thread together to effectively reduce a dataset. The most inclusive threads and their associated document attachments are maintained, and non-inclusive or duplicative thread components are set aside. However, as data volumes continue to grow and outpace deadlines for legal proceedings, practitioners often look to incorporate multiple cost-effective and defensible technology solutions to further reduce or otherwise accelerate document review and classification. One such technology is predictive modeling – known in the legal industry as ‘predictive coding’ or ‘Technology Assisted Review (TAR)’ – which is a popular tool used to augment a manual document review and classification process. Like email threading, predictive coding has become more commonplace recently for its proven ability to minimize manual document classification, thus reducing the time and cost associated with this aspect of legal proceedings.In this study, we explore the performance impacts of layering predictive modeling onto an email threading reduction workflow. Generally, a predictive model is established first, and email threading is layered onto the scored output to further reduce and streamline document review. Our research evaluates a reversed workflow, where a population is initially reduced to its inclusive email threads, and this limited dataset is used to train and apply a predictive model to the larger population of documents for classification. Using classified data from four real-world legal proceedings, we compare the performance impact of email threading on predictive modeling by 1) training a model using all positive and negative examples, 2) training a model using positive and negative examples from only inclusive email threads, and 3) training a model using positive examples from only inclusive email threads, but negative examples from all emails. The results of our research provide thoughtful, empirical insights for legal practitioners to review when exploring the deployment of both email threading and predictive modeling into a single, cohesive document classification strategy for their legal proceedings.

The costs involved in manually reviewing documents in legal civil litigations have grown dramatically as more and more information is stored electronically. As a result, the document review process can require an extraordinary dedication of resources. In construction litigations, quickly finding supporting documentation in a delay dispute is critical to the success of a matter. Identifying relevant delay-related communications and supporting documentation has historically been expensive and time consuming. Using machine learning technologies, respondents can be more comprehensive in their assessment of the data requiring review to respond to the claim in time. Explainable machine learning is an active machine learning research area, and in an explainable machine learning system, predictions generated from a machine learning model are explainable and human understandable. In delay dispute ‘document review’ scenarios, a document can be identified as delay-related, as long as one or more of the text snippets in a document are deemed delay-related. In these scenarios, if these delay-related snippets can be located, then attorneys could easily evaluate the model’s decision. The authors of this paper propose an approach for accurately identifying rationales and an approach for boosting document classification accuracy using delay-related snippets and their applications in construction delay disputes. The authors conducted experiments using data from a few real world delay dispute matters and the results from these experiments show that the proposed approaches have the potential to significantly advance the application of text classification in document review in construction delay dispute matters.

The increased integration of Large Language Models (LLMs) across industry sectors is enabling domain experts with new text classification optimization methods. These LLMs are pretrained on exceedingly large amounts of data; however, practitioners can perform additional training, or “fine-tuning,” to improve their text classifier’s results for their own use cases. This paper presents a series of experiments comparing a standard, pretrained DistilBERT model and a fine-tuned DistilBERT model, both leveraged for the downstream NLP task of text classification. Tuning the model using domain-specific data from real-world legal matters suggests fine-tuning improves the performance of LLM text classifiers.To evaluate the performance of text classification models, using these two Large Language Models, we employed two distinct approaches that 1) score a whole document’s text for prediction and 2) score snippets (sentence-level components of a document) of text for prediction. When comparing the two approaches we found that one prediction method outperforms the other, depending on the use case.

In the domain of natural language processing and text analysis, the Bidirectional Encoder Representations from Transformers (BERT) has emerged as a powerful tool for discerning the intricate nuances of textual data. Nonetheless, BERT’s inherent token limit of 512 tokens presents a noteworthy challenge when confronted with exceedingly long documents. Lengthy documents are commonly encountered during legal document review and often exceed the 512-token constraint. In response to this challenge, this study empirically compares two distinct applications of BERT, leveraging real-world, construction industry legal data. The approach compares applications of BERT to the entire document and on segmented text portions from each document. In the latter approach, the highest-scoring text segment from each document represents the document’s score. This research offers practical insights for effectively utilizing BERT in scenarios where document length exceeds the token limit. Our results allow practitioners and researchers to make informed choices when confronted with documents of significant length, thus contributing to a more effective and insightful application of BERT for text analysis.

In this paper, we discuss the application of visual object-detection analysis on an information governance business use case. Specifically, we discuss how handwriting detection can serve as the centerpiece of a workflow where signatures are detected in PDF documents, and the number of detected signatures is utilized as a method of singling out countersigned and fully executed contracts. We outline the business case and the recommended workflow involving final validation using an eDiscovery platform and provide statistics from a real-world project to illustrate the potential effectiveness of the solution.

In recent years, transformer-based, large-scale language models have greatly advanced deep learning in NLP tasks. These models allow transfer learning to be performed on NLP in similar methods to what was previously done for computer vision. Two of the latest models are DistilBERT and Longformer -- the former is a distilled version of BERT, which enables faster training and inferencing on off-cloud servers, and the latter offers capability of working with long texts. In this paper, we study empirical comparisons of the effectiveness of the two deep learning methods along with a logistical regression method for text classification, using three real-world datasets from legal document reviews. The study shows that Longformer performs better (up to 10%) than or at par with the other two methods. Cross-dataset evaluation is leveraged as well to validate the performance of Longformer as a viable method when labeled data is not available.

During the past decade breakthroughs in GPU hardware and deep neural networks technologies have revolutionized the field of computer vision, making image analytical potentials accessible to a range of real-world applications. Technology Assisted Review (TAR) in electronic discovery though traditionally has dominantly dealt with textual content, is witnessing a rising need to incorporate multimedia content in the scope. We have developed innovative image analytics applications for TAR in the past years, such as image classification, image clustering, and object detection, etc. In this paper, we discuss the use of image clustering applications to facilitate TAR based on our experiences in serving clients. We describe our general workflow on leveraging image clustering in tasks and use statistics from real projects to showcase the effectiveness of using image clustering in TAR. We also summarize lessons learned and best practices on using image clustering in TAR.