Li Zhou, Taelin Karidi, Wanlong Liu, Nicolas Garneau, Yong Cao, Wenyu Chen, Haizhou Li, Daniel Hershcovich. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Knowledge claims are abundant in the literature on large language models (LLMs); but can we say that GPT-4 truly "knows" the Earth is round?To address this question, we review standard definitions of knowledge in epistemology and we formalize interpretations applicable to LLMs.In doing so, we identify inconsistencies and gaps in how current NLP research conceptualizes knowledge with respect to epistemological frameworks.Additionally, we conduct a survey of 100 professional philosophers and computer scientists to compare their preferences in knowledge definitions and their views on whether LLMs can really be said to know.Finally, we suggest evaluation protocols for testing knowledge in accordance to the most relevant definitions.

The task of legal precedent retrieval is essential yet challenging for legal professionals, as it involves identifying relevant past cases that can inform current legal decisions. Building on previous work that integrates citation networks and text similarity analysis, we apply these techniques to a dataset comprising paragraphs from cases decided by the Court of Justice of the European Union (CJEU). While paragraph citation retrieval is way more challenging than case citation retrieval, we show that a careful combination of network and text signals improves computational efficiency without sacrificing performances. More precisely, our experiments first reveal the limitations of network analysis at the paragraph level due to the sparse connectivity of the data. We then explore a novel approach to this task by combining network analysis at the case level and natural language processing at the paragraph level, which we refer to as “pincites”.

In this work, we conduct a detailed analysis on the performance of legal-oriented pretrained language models (PLMs).We examine the interplay between their original objective, acquired knowledge, and legal language understanding capacities which we define as the upstream, probing, and downstream performance, respectively.We consider not only the models' size but also the pre-training corpora used as important dimensions in our study.To this end, we release a multinational English legal corpus (LeXFiles) and a legal knowledge probing benchmark (LegalLAMA) to facilitate training and detailed analysis of legal-oriented PLMs.We release two new legal PLMs trained on LeXFiles and evaluate them alongside others on LegalLAMA and LexGLUE.We find that probing performance strongly correlates with upstream performance in related legal topics.On the other hand, downstream performance is mainly driven by the model's size and prior legal knowledge which can be estimated by upstream and probing performance.Based on these findings, we can conclude that both dimensions are important for those seeking the development of domain-specific PLMs.

In this paper, we introduce a new beam search algorithm that improves the generalization of neural generators to unseen examples, especially in low-resource data-to-text settings.Our algorithm aims to reduce the number of omissions and hallucinations during the decoding process.For this purpose, it relies on two regression models to explicitly characterize factual errors.We explain how to create a new dataset to train these models given an original training set of less than a thousand data points.We apply our approach in the low-resource, legal setting using the French Plum2Text dataset, as well as in English using WebNLG.We observe in our experiment that this combination improves the faithfulness of pre-trained neural text generators using both human and automatic evaluation.Moreover, our approach offers a level of interpretability by predicting the number of omissions and hallucinations present in a given generation with respect to the input data.Finally, we visualize our algorithm's exploration of the hypothesis space at different steps during the decoding process.

Case law citation prediction, i.e., predicting what historical cases are relevant for your current case, can assist legal discovery and decision-making, but legal documents are long, and often only parts of them are relevant for a particular use case. We therefore reframe case law citation prediction as a paragraph-to-paragraph citation task, introduce a new dataset, and train and evaluate new models. We also evaluate our models qualitatively. Our resources provide a first step toward discovering citation patterns and modeling legal rules in EU law from precedent documents.

Language encoders encode words and phrases in ways that capture their local semantic relatedness, but are known to be globally inconsistent. Global inconsistency can seemingly be corrected for, in part, by leveraging signals from knowledge bases, but previous results are partial and limited to monolingual English encoders. We extract a large-scale multilingual, multi-word analogy dataset from Wikidata for diagnosing and correcting for global inconsistencies, and then implement a four-way Siamese BERT architecture for grounding multilingual BERT (mBERT) in Wikidata through analogy training. We show that analogy training not only improves the global consistency of mBERT, as well as the isomorphism of language-specific subspaces, but also leads to consistent gains on downstream tasks such as bilingual dictionary induction and sentence retrieval.

In this paper, we introduce a new French Data-to-Text (D2T) dataset in the legal domain: Plum2Text1. It is made out of plumitifs (docket files) - descriptions pairs that are derived from publicly available documents issued by Canadian criminal courts. The development of Plum2Text is primarily intended to train statistical natural language generation algorithms, in order to make the plumitifs more easily understandable for Canadian citizens. The inputs and outputs of the dataset are unique: on the data side, the values of the table contain long pieces of textual utterance, and on the text side (or reference), it most often consists of a paraphrase of the table values. We describe how we curated the plumitif-description associations by introducing an annotation tool and a methodology specific to the D2T natural language generation task. We do so by using simple yet efficient text classifiers to help the annotator leverage annotated examples during the annotation process. As a matter of privacy, we also illustrate how we are decontextualizing the descriptions.

Learning language representations is a key component in many natural language processing tasks, and their usefulness is most often challenged by specialized target domains and vocabulary. We have witnessed several neural causal language models (CLM) that learn contextual representations such as ELMo [8]. More recently, the Transformer architecture [10] has tremendously improved language representation learning, giving birth to new architectures such as BERT [4], a masked language model, pushing the state-of-the-art of natural language understanding to an unprecedented level of performance on standard benchmarks. Moreover, it has been found that Transformer-based CLM, such as GPT [9], are excellent feature extractors as well as being impressive text generators. BART [7], an architecture combining the backbone of both BERT and GTP proved to be particularly effective at generating text while being competitive in comprehension tasks. BARThez, the French version of BART, was recently introduced as a pre-trained model on a very large monolingual French corpus [6]. In this paper, we introduce CriminelBART, a fine-tuned version of BARThez specialized for criminal law using a French Canadian corpus of legal judgments, and we evaluate its performance on different tasks.

We hereby present our submission to the Shared Task in Evaluating Accuracy (Reiter and Thomson, 2020) at the INLG 2021 Conference.Our evaluation protocol relies on three main components; rules and text classifiers that pre-annotate the dataset, a human annotator that validates the pre-annotations, and a web interface that facilitates this validation.Our submission consists in fact of two submissions; we first analyze solely the performance of the rules and classifiers (pre-annotations), and then the human evaluation aided by the former pre-annotations using the web interface (hybrid).The code for the web interface and the classifiers is publicly available 1 .

Plumitifs (dockets) were initially a tool for law clerks.Nowadays, they are used as summaries presenting all the steps of a judicial case.Information concerning parties' identity, jurisdiction in charge of administering the case, and some information relating to the nature and the course of the preceding are available through plumitifs.They are publicly accessible but barely understandable; they are written using abbreviations and referring to provisions from the Criminal Code of Canada, which makes them hard to reason about.In this paper, we propose a simple yet efficient multi-source language generation architecture that leverages both the plumitif and the Criminal Code's content to generate intelligible plumitifs descriptions.It goes without saying that ethical considerations rise with these sensitive documents made readable and available at scale, legitimate concerns that we address in this paper.This is, to the best of our knowledge, the first application of plumitifs descriptions generation made available for French speakers along with an ethical discussion about the topic.

In this paper, we reproduce the experiments of Artetxe et al. (2018b) regarding the robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. We show that the reproduction of their method is indeed feasible with some minor assumptions. We further investigate the robustness of their model by introducing four new languages that are less similar to English than the ones proposed by the original paper. In order to assess the stability of their model, we also conduct a grid search over sensible hyperparameters. We then propose key recommendations that apply to any research project in order to deliver fully reproducible research.