Due to their ability to process long and complex contexts, LLMs can offer key benefits to the Legal domain, but their adoption has been hindered by their tendency to generate unfaithful, ungrounded, or hallucinatory outputs. While Retrieval-Augmented Generation offers a promising solution by grounding generations in external knowledge, it offers no guarantee that the provided context will be effectively integrated. To address this, context-aware decoding strategies have been proposed to amplify the influence of relevant context, but they usually do not explicitly enforce faithfulness to the context. In this work, we introduce Confidence-guided Copy-based Decoding for Legal Text Generation (CoCoLex)-a decoding strategy that dynamically interpolates the model produced vocabulary distribution with a distribution derived based on copying from the context. CoCoLex encourages direct copying based on the model's confidence, ensuring greater fidelity to the source. Experimental results on five legal benchmarks demonstrate that CoCoLex outperforms existing context-aware decoding methods, particularly in long-form generation tasks.

In legal decisions, split votes (SV) occur when judges cannot reach a unanimous decision, posing a difficulty for lawyers who must navigate diverse legal arguments and opinions.In high-stakes domains, understanding the alignment of perceived difficulty between humans and AI systems is crucial to build trust.However, existing NLP calibration methods focus on a classifier's awareness of predictive performance, measured against the human majority class, overlooking inherent human label variation (HLV).This paper explores split votes as naturally observable human disagreement and value pluralism.We collect judges' vote distributions from the European Court of Human Rights (ECHR), and present SV-ECHR 1 a case outcome classification (COC) dataset with SV information.We build a taxonomy of disagreement with SV-specific subcategories.We further assess the alignment of perceived difficulty between models and humans, as well as confidence-and human-calibration of COC models.We observe limited alignment with the judge vote distribution.To our knowledge, this is the first systematic exploration of calibration to human judgements in legal NLP.Our study underscores the necessity for further research on measuring and enhancing model calibration considering HLV in legal decision tasks.* Following Chalkidis et al. 2022a; Santosh et al. 2022 We use only the 10 most prominent ECHR articles.* https://hudoc.echr.coe.int* App B offers details on the quality assessment process.* See App D for more details of our metadata correction.* For a comprehensive understanding of each taxonomy category, we direct the reader to Xu et al. 2023b

Recognizing vulnerability is crucial for understanding and implementing targeted support to empower individuals in need.This is especially important at the European Court of Human Rights (ECtHR), where the court adapts convention standards to meet actual individual needs and thus to ensure effective human rights protection.However, the concept of vulnerability remains elusive at the ECtHR and no prior NLP research has dealt with it.To enable future work in this area, we present VECHR, a novel expert-annotated multi-label dataset comprised of vulnerability type classification and explanation rationale.We benchmark the performance of state-of-the-art models on VECHR from both the prediction and explainability perspective.Our results demonstrate the challenging nature of the task with lower prediction performance and limited agreement between models and experts.We analyze the robustness of these models in dealing with out-of-domain (OOD) data and observe limited overall performance.Our dataset poses unique challenges offering a significant room for improvement regarding performance, explainability, and robustness.

In legal NLP, Case Outcome Classification (COC) must not only be accurate but also trustworthy and explainable.Existing work in explainable COC has been limited to annotations by a single expert.However, it is well-known that lawyers may disagree in their assessment of case facts.We hence collect a novel dataset RAVE: Rationale Variation in ECHR 1 , which is obtained from two experts in the domain of international human rights law, for whom we observe weak agreement.We study their disagreements and build a two-level task-independent taxonomy, supplemented with COC-specific subcategories.We quantitatively assess different taxonomy categories and find that disagreements mainly stem from underspecification of the legal context, which poses challenges given the typically limited granularity and noise in COC metadata.To our knowledge, this is the first work in the legal NLP that focuses on building a taxonomy over human label variation.We further assess the explainablility of state-of-the-art COC models on RAVE and observe limited agreement between models and experts.Overall, our case study reveals hitherto underappreciated complexities in creating benchmark datasets in legal NLP that revolve around identifying aspects of a case's facts supposedly relevant to its outcome.

This work demonstrates that Legal Judgement Prediction systems without expert-informed adjustments can be vulnerable to shallow, distracting surface signals that arise from corpus construction, case distribution, and confounding factors.To mitigate this, we use domain expertise to strategically identify statistically predictive but legally irrelevant information.We adopt adversarial training to prevent the system from relying on it.We evaluate our deconfounded models by employing interpretability techniques and comparing to expert annotations.Quantitative experiments and qualitative analysis show that our deconfounded model consistently aligns better with expert rationales than baselines trained for prediction only.We further contribute a set of reference expert annotations to the validation and testing partitions of an existing benchmark dataset of European Court of Human Rights cases.