In this work, we conduct a detailed analysis on the performance of legal-oriented pretrained language models (PLMs).We examine the interplay between their original objective, acquired knowledge, and legal language understanding capacities which we define as the upstream, probing, and downstream performance, respectively.We consider not only the models' size but also the pre-training corpora used as important dimensions in our study.To this end, we release a multinational English legal corpus (LeXFiles) and a legal knowledge probing benchmark (LegalLAMA) to facilitate training and detailed analysis of legal-oriented PLMs.We release two new legal PLMs trained on LeXFiles and evaluate them alongside others on LegalLAMA and LexGLUE.We find that probing performance strongly correlates with upstream performance in related legal topics.On the other hand, downstream performance is mainly driven by the model's size and prior legal knowledge which can be estimated by upstream and probing performance.Based on these findings, we can conclude that both dimensions are important for those seeking the development of domain-specific PLMs.

Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, Nikolaos Aletras. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

In this paper, we highlight the potential of n-grams as a vehicle to explore the evolution of legal language. Using the full text corpus of decisions of the U.S. Supreme Court (1946–2005), we very briefly explore the n-gram space and offer some initial results based upon our calculations.

In this paper, we compare several network representations of the corpus of United States Supreme Court decisions (1791--2005). This corpus is not only of seminal importance, but also represents a highly structured and largely self-contained body of case law. As constructed herein, nodes represent whole cases or individual "opinion units" within cases. Edges represent either citations or semantic connections. As our broader goal is to better understand American common law development, we are particularly interested in the union, intersect and compliment of these various citation networks as they offer potential insight into the time developing structure of the "web of the law."