Tunnel vision and confirmation bias can lead to miscarriages of justice. A way to avoid tunnel vision is to consider your evidence in light of more than one scenario. Alternative scenarios allow us to consider how probable each scenario is, compared to the other considered scenarios. Bayesian Networks have been proposed as a formal method for reasoning about the probability of scenarios. Specifically, alternative scenarios were modelled using Bayesian networks with a constraint node, which ensures mutual exclusivity. However, the performance of these methods in situations where not all possible alternative scenarios are modeled, the non-exhaustive setting, has not been investigated. Since it is impossible to explicitly cover everything that could possibly have happened in a model, it is important to know how these methods handle non-exhaustiveness. We evaluate four methods using an agent-based model that simulates an environment in which a crime could occur. Taking this as the ground truth, we compare different Bayesian network modeling methods on five aspects related to the quality of the representation of the ground truth as well as computational performance. We find that some methods result in disparities between the ground truth and the predicted posterior probabilities for the scenarios in a non-exhaustive setting. In an exhaustive setting, the proposed methods perform well. The construction approach that models scenarios in terms of conjunctions of events performs well in both settings.

Generative large language models as tools in the legal domain have the potential to improve the justice system. However, the reasoning behavior of current generative models is brittle and poorly understood, hence cannot be responsibly applied in the domains of law and evidence. This paper presents reasoning benchmarks that are dynamically varied, scalable in their complexity, and have formally unambiguous interpretations. In this study, we illustrate the approach on the basis of witness testimony, focusing on the underlying argument attack structure. We dynamically generate both linear and non-linear argument attack graphs of varying complexity and translate these into reasoning puzzles about witness testimony expressed in natural language. We show that state-of-the-art large language models often fail in these reasoning puzzles, already at low complexity. Obvious mistakes are made by the models, and their inconsistent performance indicates that their reasoning capabilities are brittle. Furthermore, at higher complexity, even state-of-the-art models specifically designed for reasoning make mistakes. We show the viability of using a parametrized benchmark with varying complexity to evaluate the reasoning capabilities of generative language models, which contribute to a better understanding of the limitations of the reasoning capabilities of generative models.

Scenario-based Bayesian networks (BNs) have been proposed as a tool for the rational handling of evidence. The proper evaluation of existing methods requires access to a ground truth that can be used to test the quality and usefulness of a BN model of a crime. However, that would require a full probability distribution over all relevant variables used in the model, which is in practice not available. In this paper, we use an agent-based simulation as a proxy for the ground truth for the evaluation of BN models as tools for the rational handling of evidence. We use fictional crime scenarios as a background. First, we design manually constructed BNs using existing design methods in order to model example crime scenarios. Second, we build an agent-based simulation covering the scenarios of criminal and non-criminal behavior. Third, we algorithmically determine BNs using statistics collected experimentally from the agent-based simulation that represents the ground truth. Finally, we compare the manual, scenario-based BNs to the algorithmic BNs by comparing the posterior probability distribution over outcomes of the network to the ground-truth frequency distribution over those outcomes in the simulation, across all evidence valuations. We find that both manual BNs and algorithmic BNs perform similarly well: they are good reflections of the ground truth in most of the evidence valuations. Using ABMs as a ground truth can be a tool to investigate Bayesian Networks and their design methods, especially under circumstances that are implausible in real-life criminal cases, such as full probabilistic information.

One way of reasoning with uncertainties in the context of law is to use probabilities. However, methods for reasoning about the probability of guilt in a court case requires us to specify a prior probability of guilt, which is the probability of guilt before any evidence is known. There is no accepted approach for specifying the prior probability of guilt but multiple solutions have been proposed. In this paper, we consider three approaches: a prior that is based on the population, a prior based on the number of agents that have similar opportunity as the suspect and a prior that represents a legal norm. For comparing and evaluating the approaches, we use an agent-based model as a ground truth in which all probabilities are known. With the data generated in the ground truth model, we investigate how the choice of prior influences the posterior probability of guilt for both guilty and innocent agents. Using a decision threshold, we can determine the effect of the three approaches on the rates of correct and incorrect convictions and acquittals. We find that the opportunity prior results in higher rates of both correct convictions and false convictions and requires more assumptions and access to data and knowledge than the legal prior and population prior.

Data-driven AI systems can make the right decisions for the wrong reasons, which can lead to irresponsible behavior. The rationale of such machine learning models can be evaluated and improved using a previously introduced hybrid method. This method, however, was tested using synthetic data under ideal circumstances, whereas labelled datasets in the legal domain are usually relatively small and often contain missing facts or inconsistencies. In this paper, we therefore investigate rationales under such imperfect conditions. We apply the hybrid method to machine learning models that are trained on court cases, generated from a structured representation of Article 6 of the ECHR, as designed by legal experts. We first evaluate the rationale of our models, and then improve it by creating tailored training datasets. We show that applying the rationale evaluation and improvement method can yield relevant improvements in terms of both performance and soundness of rationale, even under imperfect conditions.

-independence is a novel concept concerned with explaining the (ir)relevance of intermediate nodes for maximum a posteriori () computations in Bayesian networks. Building upon properties of -independence, we introduce and experiment with methods for finding sets of relevant nodes using both an exhaustive and a heuristic approach. Our experiments show that these properties significantly speed up run time for both approaches. In addition, we link -independence to defeasible reasoning, a type of reasoning that analyses how new evidence may invalidate an already established conclusion. Ways to present users with an explanation using -independence are also suggested.

In AI and law, systems that are designed for decision support should be explainable when pursuing justice. In order for these systems to be fair and responsible, they should make correct decisions and make them using a sound and transparent rationale. In this paper, we introduce a knowledge-driven method for model-agnostic rationale evaluation using dedicated test cases, similar to unit-testing in professional software development. We apply this new quantitative human-in-the-loop method in a machine learning experiment aimed at extracting known knowledge structures from artificial datasets from a real-life legal setting. We show that our method allows us to analyze the rationale of black box machine learning systems by assessing which rationale elements are learned or not. Furthermore, we show that the rationale can be adjusted using tailor-made training data based on the results of the rationale evaluation.

The justification of an algorithm’s outcomes is important in many domains, and in particular in the law. However, previous research has shown that machine learning systems can make the right decisions for the wrong reasons: despite high accuracies, not all of the conditions that define the domain of the training data are learned. In this study, we investigate what the system does learn, using state-of-the-art explainable AI techniques. With the use of SHAP and LIME, we are able to show which features impact the decision making process and how the impact changes with different distributions of the training data. However, our results also show that even high accuracy and good relevant feature detection are no guarantee for a sound rationale. Hence these state-of-the-art explainable AI techniques cannot be used to fully expose unsound rationales, further advocating the need for a separate method for rationale evaluation.

In this paper, we present the information graph (IG) formalism, which provides a precise account of the interplay between deductive and abductive inference and causal and evidential information, where ‘deduction’ is used for defeasible ‘forward’ inference. IGs formalise analyses performed by domain experts in the informal reasoning tools they are familiar with, such as mind maps used in crime analysis. Based on principles for reasoning with causal and evidential information given the evidence, we impose constraints on the inferences that may be performed with IGs. Our IG-formalism is intended to facilitate the construction of formal representations within AI systems by serving as an intermediary formalism between analyses performed using informal reasoning tools and formalisms that allow for formal evaluation. In this paper, we investigate the use of the IG-formalism as an intermediary formalism in facilitating Bayesian network (BN) graph construction. We propose a structured approach for automatically constructing from an IG a directed BN graph, together with qualitative constraints on the probability distribution represented by the BN. Moreover, we prove a number of formal properties of our approach and identify assumptions under which the construction of an initial BN graph can be fully automated.

In this paper, we propose the information graph (IG) formalism, which provides a precise account of the interplay between deductive and abductive inference and causal and evidential information. IGs formalise analyses performed by domain experts in the informal reasoning tools they are familiar with, such as mind maps used in crime analysis. Based on principles for reasoning with causal and evidential information given the evidence, we impose constraints on the inferences that may be performed with IGs. Moreover, we propose an argumentation formalism based on IGs that allows arguments to be formally evaluated.

Bayesian networks (BNs) are powerful tools that are increasingly being used by forensic and legal experts to reason about the uncertain conclusions that can be inferred from the evidence in a case. Although in BN construction it is good practice to document the model itself, the importance of documenting design decisions has received little attention. Such decisions, including the (possibly conflicting) reasons behind them, are important for legal experts to understand and accept probabilistic models of cases. Moreover, when disagreements arise between domain experts involved in the construction of BNs, there are no systematic means to resolve such disagreements. Therefore, we propose an approach that allows domain experts to explicitly express and capture their reasons pro and con modelling decisions using argumentation, and that resolves their disagreements as much as possible. Our approach is based on a case study, in which the argumentation structure of an actual disagreement between two forensic BN experts is analysed.

In this paper, we propose a structured approach for transforming legal arguments to a Bayesian network (BN) graph. Our approach automatically constructs a fully specified BN graph by exploiting causality information present in legal arguments. Moreover, we demonstrate that causality information in addition provides for constraining some of the probabilities involved. We show that for undercutting attacks it is necessary to distinguish between causal and evidential attacked inferences, which extends on a previously proposed solution to modelling undercutting attacks in BNs. We illustrate our approach by applying it to part of an actual legal case, namely the Sacco and Vanzetti legal case.

In this paper, we propose a way to derive constraints for a Bayesian Network from structured arguments. Argumentation and Bayesian networks can both be considered decision support techniques, but are typically used by experts with different backgrounds. Bayesian network experts have the mathematical skills to understand and construct such networks, but lack expertise in the application domain; domain experts may feel more comfortable with argumentation approaches. Our proposed method allows us to check Bayesian networks given arguments constructed for the same problem, and also allows for transforming arguments into a Bayesian network structure, thereby facilitating Bayesian network construction.

A Bayesian network is a concise representation of a joint probability distribution, which can be used to compute any probability of interest for the represented distribution. Credal networks were introduced to cope with the inevitable inaccuracies in the parametrisation of such a network. Where a Bayesian network is parametrised by defining unique local distributions, in a credal network sets of local distributions are given. From a credal network, lower and upper probabilities can be inferred. Such inference, however, is often problematic since it may require a number of Bayesian network computations exponential in the number of credal sets. In this paper we propose a preprocessing step that is able to reduce this complexity. We use sensitivity functions to show that for some classes of parameter in Bayesian networks the qualitative effect of a parameter change on an outcome probability of interest is independent of the exact numerical specification. We then argue that credal sets associated with such parameters can be replaced by a single distribution.

Over the last decades the rise of forensic sciences has led to an increase in the availability of statistical evidence. Reasoning about statistics and probabilities in a forensic science setting can be a precarious exercise, especially so when independencies between variables are involved. To facilitate the correct explanation of such evidence we investigate how argumentation models can help in the interpretation of statistical information. In this paper we focus on the connection between argumentation models and Bayesian belief networks, the latter being a common model to represent and reason with complex probabilistic information. We introduce the notion of a support graph as an intermediate structure between Bayesian networks and argumentation models. A support graph disentangles the complicating graphical properties of a Bayesian network and enhances its intuitive interpretation. Moreover, we show that this model can provide a suitable template for argumentative analysis. Especially in the context of legal reasoning, the correct treatment of statistical evidence is important.

Reasoning about statistics and probabilities can, when not treated with cautiousness, lead to reasoning errors. Over the last decades the rise of forensic sciences has led to an increase in the availability of statistical evidence. To facilitate the correct explanation of such evidence we investigate how argumentation models can help in the interpretation of statistical information. Uncertainties are by forensic experts often expressed numerically, but lawyers, judges and other legal experts have notorious difficulty interpreting these results [3, 1, 2, 5]. In this demonstration of our main paper [6] we focus on the connection between formal models of argumentation and Bayesian belief networks (BNs). We use BNs because they are a well-known model to represent and reason with complex probabilistic information. We introduce the notion of a support graph as an intermediate structure between Bayesian networks and argumentation models. A support graph captures the inferences modelled in a Bayesian network but disentangles the complicating graphical properties of such models and instead emphasises its intuitive understanding. Moreover, we show that this intermediate model can function as a template to generate different arguments based on the data.

In a criminal trial, a judge or jury needs to reach a conclusion about 'what happened' based on the available evidence. Often this includes probabilistic evidence. Whereas Bayesian networks form a good tool for analysing evidence probabilistically, simply presenting the outcome of the network to a judge or jury does not allow them to make an informed decision. In this paper, we propose to combine Bayesian networks with a narrative approach to reasoning with legal evidence, the result of which allows a juror to reason with alternative scenarios while also incorporating probabilistic information. The proposed method aids both the construction and the understanding of Bayesian networks, using scenario schemes. We make three distinct contributions: (1) we propose to use scenario schemes to aid the construction of Bayesian networks, (2) we propose a method for producing scenarios in text form from the resulting networks and (3) we propose a format for reporting the alternative scenarios and their relations to the evidence (including strength).

Legal reasoning about evidence can be a precarious exercise, in particular when statistics are involved. A number of recent miscarriages of justice have provoked a scientific interest in formal models of legal evidence. Two such models are presented by Bayesian networks (BNs) and argumentation. A limitation of argumentation is that it is difficult to embed probabilities. BNs, on the other hand, are probabilistic by nature. A disadvantage of BNs is that it can be hard to explain what is modelled and how the results came about. Assuming that a forensic expert presents evidence in a way that is either already a BN or expressed in terms that easily map to a simple BN, we may wish to express the same information in argumentative terms. We address this issue by translating Bayesian networks to arguments. We do this by means of an intermediate structure, called a support graph, which represents the variables from the Bayesian network, maintaining independence information in the network, but connected in a way that more closely resembles argumentation. In the current paper we test the support graph method on a Bayesian network from the literature. We argue that the resulting support graph adequately captures the possible arguments about the represented case. In addition, we highlight strengths and limitations of the method that are revealed by this case study.

Legal reasoning with evidence can be a challenging task. We study the relation between two formal approaches that can aid the construction of legal proof: argumentation and Bayesian networks (BNs). Argument schemes are used to describe recurring patterns in argumentation. Critical questions for many argument schemes have been identified. Due to the increased use of statistical forensic evidence in court it may be advantageous to consider probabilistic models of legal evidence. In this paper we show how argument schemes and critical questions can be modelled in the graphical structure of a Bayesian network. We propose a method that integrates advantages from other methods in the literature.

Bayesian networks have gained popularity as a probabilistic tool for reasoning with legal evidence. However, two common difficulties are (1) the construction and (2) the understanding of a network. In previous work, we proposed to use narrative tools and in particular scenario schemes to assist the construction and the understanding of Bayesian networks for legal cases. We proposed a construction method and a reporting format for explaining or understanding the network. The quality of a scenario, which plays an important role in the narrative approach to evidential reasoning, was not yet included in this method. In this paper, we provide a discussion of what constitutes the quality of a scenario, in terms of the narrative concepts of completeness, consistency and plausibility. We propose a probabilistic interpretation of these concepts, and show how they can be incorporated in our previously proposed method. We also illustrate with an example how these concepts concerning scenario quality can be used to explain or understand a Bayesian network.

This demonstration shows how arguments, formalised in a well defined framework, can be automatically constructed from a given Bayesian network.

One-dimensional Bayesian network classifiers (OBCs) are popular tools for classification [2]. An OBC is a Bayesian network [4] consisting of just a single class variable and several feature variables. Multi-dimensional Bayesian network classifiers (MBCs) were introduced to generalise OBCs to multiple class variables [1, 6]. Classification performance of OBCs is known to be rather good. Experimental results that support this observation were substantiated by a study of the sensitivity properties of naive OBCs [5]. In this paper we investigate the sensitivity of MBCs. We present sensitivity functions for the outcome probabilities of interest of an MBC and use these functions to study the sensitivity value. This value captures the sensitivity of an output probability to small changes in a parameter. We compare MBCs to OBCs in this respect and conclude that an MBC will on average be even more robust to parameter changes than an OBC.

Recent developments in the forensic sciences have confronted the field of legal reasoning with the new challenge of reasoning under uncertainty. Forensic results come with uncertainty and are described in terms of likelihood ratios and random match probabilities. The legal field is unfamiliar with numerical valuations of evidence, which has led to confusion and in some cases to serious miscarriages of justice. The cases of Lucia de B. in the Netherlands and Sally Clark in the UK are infamous examples where probabilistic reasoning has gone wrong with dramatic consequences. One way of structuring probabilistic information is in Bayesian networks (BNs). In this paper we explore a new method to identify legal arguments in forensic BNs. This establishes a formal connection between probabilistic and argumentative reasoning. Developing such a method is ultimately aimed at supporting legal experts in their decision making process.

In order to make an informed decision in a criminal trial, conclusions about what may have happened need to be derived from the available evidence. Recently, Bayesian networks have gained popularity as a probabilistic tool for reasoning with evidence. However, in order to make sense of a conclusion drawn from a Bayesian network, a juror needs to understand the context. In this paper, we propose to extract scenarios from a Bayesian network to form the context for the results of computations in that network. We interpret the narrative concepts of scenario schemes, local coherence and global coherence in terms of probabilities. These allow us to present an algorithm that takes the most probable configuration of variables of interest, computed from the Bayesian network, and forms a coherent scenario as a context for these variables. This way, we take advantage of the calculations in a Bayesian network, as well as the global perspective of narratives.

Legal cases involve reasoning with evidence and with the development of a software support tool in mind, a formal foundation for evidential reasoning is required. Three approaches to evidential reasoning have been prominent in the literature: argumentation, narrative and probabilistic reasoning. In this paper a combination of the latter two is proposed.

Legal reasoning can be approached from various perspectives, traditionally argumentation, probability and narrative. The communication between forensic experts and a judge or jury would benefit from an integration of these approaches. In previous papers we worked on the connection between the narrative and the probabilistic approach. We developed techniques for representing crime scenarios in a Bayesian network. But for complex cases, the construction of a Bayesian network structure using these techniques remained a cumbersome task.

Sensitivity analysis in hidden Markov models (HMMs) is usually performed by means of a perturbation analysis where a small change is applied to the model parameters, upon which the output of interest is re-computed. Recently it was shown that a simple mathematical function describes the relation between HMM parameters and an output probability of interest; this result was established by representing the HMM as a (dynamic) Bayesian network. To determine this sensitivity function, it was suggested to employ existing Bayesian network algorithms. Up till now, however, no special purpose algorithms for establishing sensitivity functions for HMMs existed. In this paper we discuss the drawbacks of computing HMM sensitivity functions, building only upon existing algorithms. We then present a new and efficient algorithm, which is specially tailored for determining sensitivity functions in HMMs.

No abstract available.