When employing neural networks in the real world, we often encounter challenges related to dataset size, data imbalance and model selection. In this paper, we investigate the benefits of incorporating information obtained from causal structures or Qualitative Models for Data Generation Processes (QM-DGP) into neural network design and training, by leveraging causal reasoning principles such as d-separation and the Markov blanket to determine relevant input variables. Through empirical analysis, we compare networks which exploit causal structures and those that do not, focusing on aspects related to dataset efficiency. Our findings show that networks which exploit causal structures require fewer samples to achieve (near-)optimal performance on average, making them more suitable in frugal learning scenarios. We also propose a causally-decomposed neural network architecture based on causal structural information and show that it is more data efficient than its fully-connected counterpart. Our results highlight the practical advantage of using causal reasoning in neural network design, particularly in settings where data efficiency plays a crucial role.

This paper proposes an improved analysis of the Qualitative Models of Data Generating Processes (QM-DGP). The approach supports (i) determination of the complexity of a Machine Learning problem and (ii) a coarse determination of the quantities of training data that are needed to train good quality models. Compared to the previously published approach to the QM-DGP analysis, this paper introduces a more thorough and theoretically sound treatment of the learning complexity. Firstly, the approach provides more rigorous determination of the complexity of the data generating processes (DGP). Secondly, the determination of the learning complexity and the required training data volumes is based on sound statistical principles for the estimation of the distributions over categorical variables. The effectiveness of the proposed method was experimentally confirmed in controlled settings. Different ground truth models were used to sample test and training data. The approach correctly predicts the size of the training data sets for which machine learning yields models supporting classification close to Bayes Error. While the majority of the experiments were carried out on probabilistic graphical models (PGM), the experiments with Neural Networks confirmed that the QM-DGP approach is not limited to PGMs.

Test and Evaluation for verification and validation (V&V) of sensor data fusion techniques utilize methods of uncertainty analysis. The Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology identifies many attributes of metrics (i.e., semantic meaning, object metrics, and subjective quality). With the growing interest in artificial intelligence (AI) due to large data corpus access, fast compute power, and machine/deep learning (ML/DL) techniques; V&V of these methods are needed. In this paper, the enhancement of the URREF to utilize a risk assessment for decision is demonstrated towards analysis/alignment of ML/DL methods that utilize multi-modal data fusion. Evidential reasoning is considered in the use case to provide data handing reliability source and processing credibility to measure decision risk in a maritime domain awareness scenario.

ChatGPT, OpenAI’s chatbot, has gained consider-able attention since its launch in November 2022, owing to its ability to formulate articulated responses to text queries and comments relating to seemingly any conceivable subject. As impressive as the majority of interactions with ChatGPT are, this large language model has a number of acknowledged shortcomings, which in several cases, may be directly related to how ChatGPT handles uncertainty. The objective of this paper is to pave the way to formal analysis of ChatGPT uncertainty handling. To this end, the ability of the Uncertainty Representation and Reasoning Framework (URREF) ontology is assessed, to support such analysis. Elements of structured experiments for reproducible results are identified. The dataset built varies Information Criteria of Correctness, Non-specificity, Self-confidence, Relevance and Inconsistency, and the Source Criteria of Reliability, Competency and Type. ChatGPT’s answers are analyzed along Information Criteria of Correctness, Non-specificity and Self-confidence. Both generic and singular information are sequentially provided. The outcome of this preliminary study is twofold: Firstly, we validate that the experimental setup is efficient in capturing aspects of ChatGPT uncertainty handling. Secondly, we identify possible modifications to the URREF ontology that will be discussed and eventually implemented in URREF ontology Version 4.0 under development.

AI-based decision support solutions require life cycles that adequately address critical steps, such as (i) finding suitable machine learning (ML) methods for the problem at hand, (ii) preparing and executing adequate data acquisition processes and (iii) tractable evaluation of the overall solution. Understanding the data generating processes is key in achieving this. Training and test data can be seen as a result of a causal data generation process, a sampling process in which the data is collected from different sources that are influenced by multiple interdependent phenomena. This is represented by a Qualitative Model of Data Generation Processes (QM-DGP), a causal graphical model. QM-DGP facilitates analysis of the complexity of the underlying data generating processes that can inform the development of trustable ML-based solutions in multiple ways. Firstly, this analysis is the basis for the determination of the required complexity of the ML models. Secondly, it facilitates the determination of the quantities of training data supporting good learning results. Thirdly, it can provide guidance for a systematic simplification of the models, supporting tractable solutions without significantly reduced performance. The construction of QM-DGP and the analysis benefit from sound theoretical concepts, such as d-separation and I-Maps. Experimental results with simulated data indicate that the approach can be effective in predicting the required quantities of training data and the determination of the modelling complexity using different types of models.

This paper introduces a probabilistic approach for forecasting vessel trajectories. It combines a Dynamic Bayesian Network (DBN) and an auxiliary position determination process to iteratively sample future vessel positions in a scalable and computationally efficient manner. The DBN is a discrete probabilistic model of typical vessel behaviors. It is used for ancestral sampling to predict the speed and orientation of a vessel which, in turn, are used by the auxiliary process to predict the vessel's position in a discretized representation of the space. The DBN is event based and uses latent variables that efficiently encode the context influencing the dynamics of different types of vessels. The parameters of the DBN are learned in an unsupervised fashion by using the Expectation Maximization (EM) algorithm. The experiments with real world data confirm the accuracy and effectiveness of the proposed approach.

This paper presents a Bayesian tracking approach that exploits various types of context information. The filtering accuracy and precision are improved by using uncertain information about (i) the constraints on the target mobility, (ii) environmental influences on the sensor performance and (iii) typical target behaviors. The approach combines particle filters with exact Bayesian networks. The overall process is equivalent to approximate inference on elaborate dynamic Bayesian networks that systematically capture non-trivial correlations between the estimated states of the dynamic processes, the associated observations and the various factors influencing the dynamic processes. The particle filter supports reasoning about continuous dynamic processes spanning large areas, while the Bayesian networks are used for the implementation of advanced sensor models and for the fusion of uncertain data on mobility constraints. The derivation of the solution is based on the decomposability principles of Bayesian networks. The approach is illustrated with the help of a challenging wildlife protection application. A set of qualitative experiments shows the improvement in tracking performance by considering the different types of context information.

The choices of the uncertainty representations and reasoning methods have a critical impact on the development and deployment of modern fusion solutions. They influence the development effort, the quality of the resulting solutions as well as the deployment costs. However, such choices require an analysis that considers many operational and theoretical aspects. The Uncertainty Representation and Reasoning Evaluation Framework (URREF) concepts enable such an analysis. The proposed URREF -driven development approach establishes relations between the URREF criteria and evaluation subjects in the context of a development and deployment life cycle. In this way the relevant theoretical elements of the fusion techniques employed are emphasized and evaluated at various stages of the development process, facilitating informed design choices as well as systematic and tractable evaluation of complex fusion solutions. The concepts are illustrated through the assessment of a high-level fusion approach supporting estimation of the whereabouts of wildlife poachers.

The International Society of Information Fusion (ISIF) Evaluation Techniques for Uncertainty Representation Working Group (ETURWG) investigates the quantification and evaluation of all types of uncertainty regarding the inputs, reasoning and outputs of the information fusion process. The ETURWG is developing an Uncertainty Representation and Reasoning Framework (URREF) ontology for this purpose. This paper outlines a start towards the process of defining metrics for the URREF data criteria, which will align the URREF ontology with practical application. A criterion can be evaluated according to several metrics, and a metric can be applied to several criteria. As such, the ontology would have to reflect the nature of a many-to-many mapping between criteria and metrics. The main findings and suggestions of the paper advancing the use of URREF are: 1) The Weight of Information (WoI) is dependent on data criteria, which in turn depend on source criteria. 2) Criteria and metrics that apply to evidence (typically an input of the fusion system), could equally apply to the fusion system outputs or internal information, which in turn could form the inputs of another system. As such the word “Evidence” in the terms “Piece of Evidence” and “Weight of Evidence” should be replaced by the word “Information”. 3) Accuracy and precision and associated metrics are ubiquitous in the URREF ontology and can evaluate many parts of the fusion system. 4) The weight of information also assumes an important position in the ontology, as it depends on several source and data criteria.

The question addressed in this paper is “what” is to be evaluated by the Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology. We thus identify the elements composing uncertainty representation and reasoning approaches, which constitute various subjects being assessed. We distinguish between primary evaluation subjects (Uncertainty Representation and Reasoning components of the fusion algorithm), and secondary evaluation subjects (source of information, piece of information, fusion method and mathematical model). This paper proposes a list of source quality criteria to be added to the ontology and establishes formal links between the secondary and primary evaluation subjects. The key contribution of the paper is the update of the definitions of sub-criteria of the Expressiveness criterion together with suggestions for complementary concepts to be included in the ontology (type of scale, type of uncertainty expression). Conclusions are drawn to extend the work in using the expressiveness criterion for information fusion analysis.

This paper considers the accuracy of state estimation based on classification using Bayesian networks. It presents a method to localize network fragments that (i) are in a particular (rare) case responsible for a potential misclassification, or (ii) contain modeling errors that consistently cause misclassifications, even in common cases. We derive an algorithm that, within such fragments, can localize the probable cause of the misclassification. The approach is based on monitoring the Bayesian network's 'behavior' at runtime, specifically the correlation among sets of evidence. We suggest several applications for the algorithm's o utput, such as repairing or mitigating the effects of errors, or deactivating faulty information sources.

In this paper we show that causal probabilistic models can facilitate the design of robust and flexible fusion systems. Observed events resulting from stochastic causal processes can be modeled with the help of causal Bayesian networks, mathematically rigorous and compact probabilistic causal models. Bayesian networks explicitly represent conditional independence which facilitates decentralized modeling and information fusion. Starting with the theory of BNs we derive design and organization rules for distributed multi-agent systems that implement exact belief propagation without centralized configuration and fusion control. In this way we can design multi-agent fusion systems which can adapt to rapidly changing information source constellations and can efficiently process large quantities of information.

This panel position paper discusses advantages and challenges of multi agent fusion systems (MAS) with respect to the modeling flexibility and fusion reliability. We argue that the MAS paradigm in combination with rigorous modeling and inference methods can facilitate design of theoretically and technically sound fusion systems. This is illustrated with the help of a MAS approach to Bayesian fusion which supports robust and efficient situation assessment in crisis management settings. However, the MAS paradigm might be ill-suited for certain domains and information fusion theories. In addition, implementation of multi agent fusion systems for real world applications can be theoretically as well as technically very challenging. We identify several research topics which address these challenges.

This paper discusses the properties of Bayesian networks (BNs) in the context of accurate state estimation. We focus on a relevant class of problems where state estimation can be viewed as a classification of possible states based on the fusion of heterogeneous and noisy information. We introduce the inference meta model (IMM), a coarse runtime perspective on the inference processes which facilitates the analysis of the state estimation with BNs. By making coarse and realistic assumptions, we show that such inference can be very robust and has asymptotic properties regarding the fusion accuracy, even if we use models and evidence associated with significant uncertainties. Moreover, the IMM provides guidance for the development of (i) robust fusion systems and (ii) methods for runtime detection of potentially misleading fusion results

Distributed perception networks (DPN) are a MAS approach to large scale fusion of heterogeneous and noisy information. DPN agents can establish meaningful information filtering channels between the relevant information sources and the decision makers. Through specification of high level concepts, DPN agent organizations generate distributed Bayesian networks, which provide mappings between the observed symptoms and the hypotheses relevant to the decision making. In addition, DPNs support robust distributed inference as well as decentralized probabilistic resource allocation.