In this paper, we introduce QABISAR, a novel framework for statutory article retrieval, to overcome the semantic mismatch problem when modeling each query-article pair in isolation, making it hard to learn representation that can effectively capture multi-faceted information. QABISAR leverages bipartite interactions between queries and articles to capture diverse aspects inherent in them. Further, we employ knowledge distillation to transfer enriched query representations from the graph network into the query bi-encoder, to capture the rich semantics present in the graph representations, despite absence of graph-based supervision for unseen queries during inference. Our experiments on a real-world expert-annotated dataset demonstrate its effectiveness.

Class imbalance poses a significant challenge in many legal text classification tasks that has remained mostly unaddressed. In the field of computer vision, in contrast, various loss functions have been developed to tackle the issue. Our study evaluates the proposed loss functions as possible drop-in solutions for transformer models pre-trained on legal data. We conduct extensive experiments that include the training of over 860 LexLM RoBERTa-large and PoL-BERT-large models for up to 20 epochs on the popular ECHR and Unfair-ToS datasets. Our results highlight that via minimal modifications to existing fine-tuning pipelines, custom loss functions can improve model performance and training behavior. We also show that while standard binary cross entropy paired with weighted random sampling leads to an increase in training time, it can be an effective tool that does not need hyperparameter tuning.

This study investigates the challenges posed by the dynamic nature of legal multi-label text classification tasks, where legal concepts evolve over time.Existing models often overlook the temporal dimension in their training process, leading to suboptimal performance of those models over time, as they treat training data as a single homogeneous block.To address this, we introduce ChronosLex, an incremental training paradigm that trains models on chronological splits, preserving the temporal order of the data.However, this incremental approach raises concerns about overfitting to recent data, prompting an assessment of mitigation strategies using continual learning and temporal invariant methods.Our experimental results over six legal multi-label text classification datasets reveal that continual learning methods prove effective in preventing overfitting thereby enhancing temporal generalizability, while temporal invariant methods struggle to capture these dynamics of temporal shifts.

In legal decisions, split votes (SV) occur when judges cannot reach a unanimous decision, posing a difficulty for lawyers who must navigate diverse legal arguments and opinions.In high-stakes domains, understanding the alignment of perceived difficulty between humans and AI systems is crucial to build trust.However, existing NLP calibration methods focus on a classifier's awareness of predictive performance, measured against the human majority class, overlooking inherent human label variation (HLV).This paper explores split votes as naturally observable human disagreement and value pluralism.We collect judges' vote distributions from the European Court of Human Rights (ECHR), and present SV-ECHR 1 a case outcome classification (COC) dataset with SV information.We build a taxonomy of disagreement with SV-specific subcategories.We further assess the alignment of perceived difficulty between models and humans, as well as confidence-and human-calibration of COC models.We observe limited alignment with the judge vote distribution.To our knowledge, this is the first systematic exploration of calibration to human judgements in legal NLP.Our study underscores the necessity for further research on measuring and enhancing model calibration considering HLV in legal decision tasks.* Following Chalkidis et al. 2022a; Santosh et al. 2022 We use only the 10 most prominent ECHR articles.* https://hudoc.echr.coe.int* App B offers details on the quality assessment process.* See App D for more details of our metadata correction.* For a comprehensive understanding of each taxonomy category, we direct the reader to Xu et al. 2023b

Legal professionals frequently encounter long legal judgments that hold critical insights for their work. While recent advances have led to automated summarization solutions for legal documents, they typically provide generic summaries, which may not meet the diverse information needs of users. To address this gap, we introduce LexAbSumm, a novel dataset designed for aspect-based summarization of legal case decisions, sourced from the European Court of Human Rights jurisdiction. We evaluate several abstractive summarization models tailored for longer documents on LexAbSumm, revealing a challenge in conditioning these models to produce aspect-specific summaries. We release LexAbSum to facilitate research in aspect-based summarization for legal domain.

The assessment of explainability in Legal Judgement Prediction (LJP) systems is of paramount importance in building trustworthy and transparent systems, particularly considering the reliance of these systems on factors that may lack legal relevance or involve sensitive attributes. This study delves into the realm of explainability and fairness in LJP models, utilizing Swiss Judgement Prediction (SJP), the only available multilingual LJP dataset. We curate a comprehensive collection of rationales that ‘support’ and ‘oppose’ judgement from legal experts for 108 cases in German, French, and Italian. By employing an occlusion-based explainability approach, we evaluate the explainability performance of state-of-the-art monolingual and multilingual BERT-based LJP models, as well as models developed with techniques such as data augmentation and cross-lingual transfer, which demonstrated prediction performance improvement. Notably, our findings reveal that improved prediction performance does not necessarily correspond to enhanced explainability performance, underscoring the significance of evaluating models from an explainability perspective. Additionally, we introduce a novel evaluation framework, Lower Court Insertion (LCI), which allows us to quantify the influence of lower court information on model predictions, exposing current models’ biases.

In common law jurisdictions, legal practitioners rely on precedents to construct arguments, in line with the doctrine of stare decisis. As the number of cases grow over the years, prior case retrieval (PCR) has garnered significant attention. Besides lacking real-world scale, existing PCR datasets do not simulate a realistic setting, because their queries use complete case documents while only masking references to prior cases. The query is thereby exposed to legal reasoning not yet available when constructing an argument for an undecided case as well as spurious patterns left behind by citation masks, potentially short-circuiting a comprehensive understanding of case facts and legal principles. To address these limitations, we introduce a PCR dataset based on judgements from the European Court of Human Rights (ECtHR), which explicitly separate facts from arguments and exhibit precedential practices, aiding us to develop this PCR dataset to foster systems’ comprehensive understanding. We benchmark different lexical and dense retrieval approaches with various negative sampling strategies, adapting them to deal with long text sequences using hierarchical variants. We found that difficulty-based negative sampling strategies were not effective for the PCR task, highlighting the need for investigation into domain-specific difficulty criteria. Furthermore, we observe performance of the dense models degrade with time and calls for further research into temporal adaptation of retrieval models. Additionally, we assess the influence of different views , Halsbury’s and Goodhart’s, in practice in ECtHR jurisdiction using PCR task.

Legal professionals often grapple with navigating lengthy legal judgements to pinpoint information that directly address their queries. This paper focus on this task of extracting relevant paragraphs from legal judgements based on the query. We construct a specialized dataset for this task from the European Court of Human Rights (ECtHR) using the case law guides. We assess the performance of current retrieval models in a zero-shot way and also establish fine-tuning benchmarks using various models. The results highlight the significant gap between fine-tuned and zero-shot performance, emphasizing the challenge of handling distribution shift in the legal domain. We notice that the legal pre-training handles distribution shift on the corpus side but still struggles on query side distribution shift, with unseen legal queries. We also explore various Parameter Efficient Fine-Tuning (PEFT) methods to evaluate their practicality within the context of information retrieval, shedding light on the effectiveness of different PEFT methods across diverse configurations with pre-training and model architectures influencing the choice of PEFT method.

Rhetorical Role Labeling (RRL) of legal judgments is essential for various tasks, such as case summarization, semantic search and argument mining. However, it presents challenges such as inferring sentence roles from context, interrelated roles, limited annotated data, and label imbalance. This study introduces novel techniques to enhance RRL performance by leveraging knowledge from semantically similar instances (neighbours). We explore inference-based and training-based approaches, achieving remarkable improvements in challenging macro-F1 scores. For inference-based methods, we explore interpolation techniques that bolster label predictions without re-training. While in training-based methods, we integrate prototypical learning with our novel discourse-aware contrastive method that work directly on embedding spaces. Additionally, we assess the cross-domain applicability of our methods, demonstrating their effectiveness in transferring knowledge across diverse legal domains.

In this paper, we introduce CuSINeS, a negative sampling approach to enhance the performance of Statutory Article Retrieval (SAR). CuSINeS offers three key contributions. Firstly, it employs a curriculum-based negative sampling strategy guiding the model to focus on easier negatives initially and progressively tackle more difficult ones. Secondly, it leverages the hierarchical and sequential information derived from the structural organization of statutes to evaluate the difficulty of samples. Lastly, it introduces a dynamic semantic difficulty assessment using the being-trained model itself, surpassing conventional static methods like BM25, adapting the negatives to the model’s evolving competence. Experimental results on a real-world expert-annotated SAR dataset validate the effectiveness of CuSINeS across four different baselines, demonstrating its versatility.

Santosh T.y.s.s, Vatsal Venkatkrishna, Saptarshi Ghosh, Matthias Grabmair. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

We report on an experiment in case outcome classification on European Court of Human Rights cases where our model first learns to identify the convention articles allegedly violated by the state from case facts descriptions, and subsequently uses that information to classify whether the court finds a violation of those articles.We assess the dependency between these two tasks at the feature and outcome level.Furthermore, we leverage a hierarchical contrastive loss to pull together article-specific representations of cases at the higher level, leading to distinctive article clusters.The cases in each article cluster are further pulled closer based on their outcome, leading to sub-clusters of cases with similar outcomes.Our experiment results demonstrate that, given a static pre-trained encoder, our models produce a small but consistent improvement in classification performance over single-task and joint models without contrastive loss.

Recognizing vulnerability is crucial for understanding and implementing targeted support to empower individuals in need.This is especially important at the European Court of Human Rights (ECtHR), where the court adapts convention standards to meet actual individual needs and thus to ensure effective human rights protection.However, the concept of vulnerability remains elusive at the ECtHR and no prior NLP research has dealt with it.To enable future work in this area, we present VECHR, a novel expert-annotated multi-label dataset comprised of vulnerability type classification and explanation rationale.We benchmark the performance of state-of-the-art models on VECHR from both the prediction and explainability perspective.Our results demonstrate the challenging nature of the task with lower prediction performance and limited agreement between models and experts.We analyze the robustness of these models in dealing with out-of-domain (OOD) data and observe limited overall performance.Our dataset poses unique challenges offering a significant room for improvement regarding performance, explainability, and robustness.

In legal NLP, Case Outcome Classification (COC) must not only be accurate but also trustworthy and explainable.Existing work in explainable COC has been limited to annotations by a single expert.However, it is well-known that lawyers may disagree in their assessment of case facts.We hence collect a novel dataset RAVE: Rationale Variation in ECHR 1 , which is obtained from two experts in the domain of international human rights law, for whom we observe weak agreement.We study their disagreements and build a two-level task-independent taxonomy, supplemented with COC-specific subcategories.We quantitatively assess different taxonomy categories and find that disagreements mainly stem from underspecification of the legal context, which poses challenges given the typically limited granularity and noise in COC metadata.To our knowledge, this is the first work in the legal NLP that focuses on building a taxonomy over human label variation.We further assess the explainablility of state-of-the-art COC models on RAVE and observe limited agreement between models and experts.Overall, our case study reveals hitherto underappreciated complexities in creating benchmark datasets in legal NLP that revolve around identifying aspects of a case's facts supposedly relevant to its outcome.

This work demonstrates that Legal Judgement Prediction systems without expert-informed adjustments can be vulnerable to shallow, distracting surface signals that arise from corpus construction, case distribution, and confounding factors.To mitigate this, we use domain expertise to strategically identify statistically predictive but legally irrelevant information.We adopt adversarial training to prevent the system from relying on it.We evaluate our deconfounded models by employing interpretability techniques and comparing to expert annotations.Quantitative experiments and qualitative analysis show that our deconfounded model consistently aligns better with expert rationales than baselines trained for prediction only.We further contribute a set of reference expert annotations to the validation and testing partitions of an existing benchmark dataset of European Court of Human Rights cases.

Lawyers and judges spend a large amount of time researching the proper legal authority to cite while drafting decisions. In this paper, we develop a citation recommendation tool that can help improve efficiency in the process of opinion drafting. We train four types of machine learning models, including a citation-list based method (collaborative filtering) and three context-based methods (text similarity, BiLSTM and RoBERTa classifiers). Our experiments show that leveraging local textual context improves recommendation, and that deep neural models achieve decent performance. We show that non-deep text-based methods benefit from access to structured case metadata, but deep models only benefit from such access when predicting from context of insufficient length. We also find that, even after extensive training, RoBERTa does not outperform a recurrent neural model, despite its benefits of pretraining. Our behavior analysis of the RoBERTa model further shows that predictive performance is stable across time and citation classes.

In this paper, we examine the use of multi-lingual sentence embeddings to transfer predictive models for functional segmentation of adjudicatory decisions across jurisdictions, legal systems (common and civil law), languages, and domains (i.e. contexts). Mechanisms for utilizing linguistic resources outside of their original context have significant potential benefits in AI & Law because differences between legal systems, languages, or traditions often block wider adoption of research outcomes. We analyze the use of Language-Agnostic Sentence Representations in sequence labeling models using Gated Recurrent Units (GRUs) that are transferable across languages. To investigate transfer between different contexts we developed an annotation scheme for functional segmentation of adjudicatory decisions. We found that models generalize beyond the contexts on which they were trained (e.g., a model trained on administrative decisions from the US can be applied to criminal law decisions from Italy). Further, we found that training the models on multiple contexts increases robustness and improves overall performance when evaluating on previously unseen contexts. Finally, we found that pooling the training data from all the contexts enhances the models' in-context performance.

We report on a pilot experiment in automatic, extractive summarization of legal cases concerning Post-traumatic Stress Disorder from the US Board of Veterans' Appeals. We hypothesize that length-constrained extractive summaries benefit from choosing among sentences that are predictive for the case outcome. We develop a novel train-attribute-mask pipeline using a CNN classifier to iteratively select predictive sentences from the case, which measurably improves prediction accuracy on partially masked decisions. We then select a subset for the summary through type classification, maximum marginal relevance, and a summarization template. We use ROUGE metrics and a qualitative survey to evaluate generated summaries along with expert-extracted and expert-drafted summaries. We show that sentence predictiveness does not reliably cover all decision-relevant aspects of a case, illustrate that lexical overlap metrics are not well suited for evaluating legal summaries, and suggest that future work should focus on case-aspect coverage.

This paper presents the Value Judgment Formalism and its experimental implementation in the VJAP system, which is capable of arguing about, and predicting outcomes of, a set of trade secret misappropriation cases. VJAP creates an argument graph for each case using argument schemes and a representation of values underlying trade secret law and effects of facts on these values. It balances effects on values in each case and analogizes it to tradeoffs in precedents. It predicts case outcomes using a confidence measure computed from the graph and generates textual legal arguments justifying its predictions. The confidence propagation uses quantitative weights learned from past cases using an iterative optimization method. Prediction performance on a limited dataset is competitive with common machine learning models. The results and VJAP's behavior are discussed in detail.

Building dialogue interfaces for realworld scenarios often entails training semantic parsers starting from zero examples.How can we build datasets that better capture the variety of ways users might phrase their queries, and what queries are actually realistic?Wang et al. (2015) proposed a method to build semantic parsing datasets by generating canonical utterances using a grammar and having crowdworkers paraphrase them into natural wording.A limitation of this approach is that it induces bias towards using similar language as the canonical utterances.In this work, we present a methodology that elicits meaningful and lexically diverse queries from users for semantic parsing tasks.Starting from a seed lexicon and a generative grammar, we pair logical forms with mixed text-image representations and ask crowdworkers to paraphrase and confirm the plausibility of the queries that they generated.We use this method to build a semantic parsing dataset from scratch for a dialog agent in a smart-home simulation.We find evidence that this dataset, which we have named SMARTHOME, is demonstrably more lexically diverse and difficult to parse than existing domain-specific semantic parsing datasets.

We report on prototype experiments expanding on prior work [2] in retrieving and ranking vaccine injury decisions using semantic information and classifying sentences as legal rules or findings about vaccine-injury causation. Our positive results include that query element coverage features and aggregate citation information using a BM25-like score can improve ranking results, and that larger amounts of annotated sentence data improve classification performance. Negative observations include that LUIMA-specific sentence features do not impact sentence classification, and that synthetic oversampling improves classification only for the sparser of the two predicted sentence types.

This paper presents first results from a proof of feasibility experiment in conceptual legal document retrieval in a particular domain (involving vaccine injury compensation). The conceptual markup of documents is done automatically using LUIMA, a law-specific semantic extraction toolbox based on the UIMA framework. The system consists of modules for automatic sub-sentence level annotation, machine learning based sentence annotation, basic retrieval using Apache Lucene and a machine learning based reranking of retrieved documents. In a leave-one-out experiment on a limited corpus, the resulting rankings scored higher for most tested queries than baseline rankings created using a commercial full-text legal information system.

In this paper we mine statutory texts for highly-specific functional information using NLP techniques and a supervised ML approach. We focus on regulatory provisions from multiple state jurisdictions (Pennsylvania and Florida), all dealing with the same general topic (i.e., public health system emergency preparedness and response). While the number of annotated provisions from any one jurisdiction is not large, we are investigating whether one can improve classification performance on one jurisdiction's statutory texts by including other jurisdictions' annotated statutory texts dealing with the same general topic. Our experiments suggest that data from one jurisdiction can be used to boost the performance of the classifiers trained for different jurisdictions.

This paper expands on the previously published value judgment formalism. The representation of situations is enhanced by introducing event progressions similar to actions in general AI planning. Using event progressions, situations can be assessed as to what facts they contain as well as what facts may ensue with some likelihood, thereby opening up a situation space. Purposive legal argumentation can be modeled using propositions and rules controlling the likelihoods of value-laden consequences. The paper expands the formalism to cover event progressions and illustrates the functionality using an example based on Young v. Hitchens.

This paper explains and illustrates in an example context how case comparison in legal case-based reasoning can be modeled in the value judgment formalism. It presents a set of argument schemes corresponding to typical moves in case-based reasoning which make use of intermediate legal concepts and their impact on the applicable values.

This paper presents preliminary results in extracting semantic information from US state public health legislative provisions using natural language processing techniques and machine learning classifiers. Challenges in the density and distribution of the data as well as the structure of the prediction task are described. Decision tree models trained on a unigram representation with TFIDF measures in most cases outperform the baselines by varying margins, leaving room for further improvement.

This paper presents a technique with which instances of argument structures in the Carneades model can be given a probabilistic semantics by translating them into Bayesian networks. The propagation of argument applicability and statement acceptability can be expressed through conditional probability tables. This translation suggests a way to extend Carneades to improve its utility for decision support in the presence of uncertainty.

This paper presents a formalism modeling legal reasoning with fact patterns and their substantive effects on legal values. It centers on the concept of a value judgment, i.e. a determination that one factual situation is preferable over another by virtue of their respective effects on values. This allows the modeling of legal sources as sets of value judgments and legal methodologies as collections of argumentation schemes. The paper briefly derives the formalism from legal theory and elaborates on its use in the context of an example of hypothetical reasoning.

This paper outlines a process model for disambiguating legal provisions in order to ease their formalization into logic. It centers around a reformulation of the provision driven by critical questioning and mandatory legal justifications.