Advancements in conversational artificial intelligence (AI) enable scalable, web-based training environments beyond traditional classroom settings to support learning. In emergency communication centers such as 9-1-1, trainees have limited opportunities to practice diverse call-handling scenarios, as it relies on role-play instruction and constrained instructor availability. We introduce SHIELD (Strengthening Human Intervention in Emergencies through Learning with Data and AI), a framework and online conversational AI system designed to support scenario-based training through interactive simulations, adaptive feedback, and learning analytics. SHIELD integrates AI-generated call simulations with data science techniques to capture fine-grained trainee interaction data, including decision-making behavior, response timing, and corrective actions. These logged interactions are analyzed to provide real-time metacognitive nudges during simulated calls and post-performance feedback through AI-assisted analytics. The framework is informed by principles of self-regulated learning to structure training across planning, execution, and reflection phases. This demo showcases SHIELD as a web-based prototype deployed for emergency call-taker training and discusses design insights from initial testing sessions conducted in collaboration with a public safety communications agency. The system also highlights potential applicability to other high-stakes operational training settings.

When employing neural networks in the real world, we often encounter challenges related to dataset size, data imbalance and model selection. In this paper, we investigate the benefits of incorporating information obtained from causal structures or Qualitative Models for Data Generation Processes (QM-DGP) into neural network design and training, by leveraging causal reasoning principles such as d-separation and the Markov blanket to determine relevant input variables. Through empirical analysis, we compare networks which exploit causal structures and those that do not, focusing on aspects related to dataset efficiency. Our findings show that networks which exploit causal structures require fewer samples to achieve (near-)optimal performance on average, making them more suitable in frugal learning scenarios. We also propose a causally-decomposed neural network architecture based on causal structural information and show that it is more data efficient than its fully-connected counterpart. Our results highlight the practical advantage of using causal reasoning in neural network design, particularly in settings where data efficiency plays a crucial role.

In today's digital landscape, phishing attacks persist as a formidable challenge, highlighting the need for robust strategies to mitigate individual risk. While advanced machine learning techniques have excelled in identifying those most susceptible to phishing, existing research has primarily focused on refining prediction accuracy rather than leveraging this understanding to mitigate risk. To bridge this gap, we present a novel counterfactual explanation approach aimed at identifying the specific traits that heighten an individual's vulnerability to phishing. Our approach integrates uncertainties and causal insights from the data generation process, producing actionable intelligence to effectively lower individual susceptibility. This enables us to tailor personalized recommendations to reduce individual's vulnerability. Through experimentation, we assess the efficacy of our methodology and demonstrate capacity to reduce susceptibility to phishing. These findings emphasize the importance of personalized interventions, arming individuals with the knowledge necessary to improve their online security protocols.

This paper proposes an improved analysis of the Qualitative Models of Data Generating Processes (QM-DGP). The approach supports (i) determination of the complexity of a Machine Learning problem and (ii) a coarse determination of the quantities of training data that are needed to train good quality models. Compared to the previously published approach to the QM-DGP analysis, this paper introduces a more thorough and theoretically sound treatment of the learning complexity. Firstly, the approach provides more rigorous determination of the complexity of the data generating processes (DGP). Secondly, the determination of the learning complexity and the required training data volumes is based on sound statistical principles for the estimation of the distributions over categorical variables. The effectiveness of the proposed method was experimentally confirmed in controlled settings. Different ground truth models were used to sample test and training data. The approach correctly predicts the size of the training data sets for which machine learning yields models supporting classification close to Bayes Error. While the majority of the experiments were carried out on probabilistic graphical models (PGM), the experiments with Neural Networks confirmed that the QM-DGP approach is not limited to PGMs.

Test and Evaluation for verification and validation (V&V) of sensor data fusion techniques utilize methods of uncertainty analysis. The Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology identifies many attributes of metrics (i.e., semantic meaning, object metrics, and subjective quality). With the growing interest in artificial intelligence (AI) due to large data corpus access, fast compute power, and machine/deep learning (ML/DL) techniques; V&V of these methods are needed. In this paper, the enhancement of the URREF to utilize a risk assessment for decision is demonstrated towards analysis/alignment of ML/DL methods that utilize multi-modal data fusion. Evidential reasoning is considered in the use case to provide data handing reliability source and processing credibility to measure decision risk in a maritime domain awareness scenario.

ChatGPT, OpenAI’s chatbot, has gained consider-able attention since its launch in November 2022, owing to its ability to formulate articulated responses to text queries and comments relating to seemingly any conceivable subject. As impressive as the majority of interactions with ChatGPT are, this large language model has a number of acknowledged shortcomings, which in several cases, may be directly related to how ChatGPT handles uncertainty. The objective of this paper is to pave the way to formal analysis of ChatGPT uncertainty handling. To this end, the ability of the Uncertainty Representation and Reasoning Framework (URREF) ontology is assessed, to support such analysis. Elements of structured experiments for reproducible results are identified. The dataset built varies Information Criteria of Correctness, Non-specificity, Self-confidence, Relevance and Inconsistency, and the Source Criteria of Reliability, Competency and Type. ChatGPT’s answers are analyzed along Information Criteria of Correctness, Non-specificity and Self-confidence. Both generic and singular information are sequentially provided. The outcome of this preliminary study is twofold: Firstly, we validate that the experimental setup is efficient in capturing aspects of ChatGPT uncertainty handling. Secondly, we identify possible modifications to the URREF ontology that will be discussed and eventually implemented in URREF ontology Version 4.0 under development.

AI-based decision support solutions require life cycles that adequately address critical steps, such as (i) finding suitable machine learning (ML) methods for the problem at hand, (ii) preparing and executing adequate data acquisition processes and (iii) tractable evaluation of the overall solution. Understanding the data generating processes is key in achieving this. Training and test data can be seen as a result of a causal data generation process, a sampling process in which the data is collected from different sources that are influenced by multiple interdependent phenomena. This is represented by a Qualitative Model of Data Generation Processes (QM-DGP), a causal graphical model. QM-DGP facilitates analysis of the complexity of the underlying data generating processes that can inform the development of trustable ML-based solutions in multiple ways. Firstly, this analysis is the basis for the determination of the required complexity of the ML models. Secondly, it facilitates the determination of the quantities of training data supporting good learning results. Thirdly, it can provide guidance for a systematic simplification of the models, supporting tractable solutions without significantly reduced performance. The construction of QM-DGP and the analysis benefit from sound theoretical concepts, such as d-separation and I-Maps. Experimental results with simulated data indicate that the approach can be effective in predicting the required quantities of training data and the determination of the modelling complexity using different types of models.

Generative models for affiliation networks condition the edges on the membership of their nodes to communities. The problem of community detection under these models is addressed by inferring the membership parameters from the network structure. Current models make several unrealistic assumptions to make the inference feasible, and are mostly designed to work on regular graphs that cannot handle multi-way connections between nodes. While the models designed for hypergraphs attempt to capture the latter, they add further strict assumptions on the structure and size of hyperedges and are usually computationally intractable for real data. This paper proposes an efficient probabilistic generative model for detecting overlapping communities that process hyperedges without any changes or restrictions on their size. Our model represents the entire state space of the hyperedges, which is exponential in the number of nodes. We develop a mathematical computation reduction scheme that reduces the inference time to linear in the volume of the hypergraph without sacrificing precision. Our experimental results validate the effectiveness and scalability of our model and demonstrate the superiority of our approach over state-of-the-art community detection methods.

Bayesian bandits using Thompson Sampling have seen increasing success in recent years. Yet existing value models (of rewards) are misspecified on many real-world problem. We demonstrate this on the User Experience Optimization (UXO) problem, providing a novel formulation as a restless, sleeping bandit with unobserved confounders plus optional stopping. Our case studies show how common misspecifications can lead to sub-optimal rewards, and we provide model extensions to address these, along with a scientific model building process practitioners can adopt or adapt to solve their own unique problems. To our knowledge, this is the first study showing the effects of overdispersion on bandit explore/exploit efficacy, tying the common notions of under- and over-confidence to over- and under-exploration, respectively. We also present the first model to demonstrate that vanishing regret and fast and consistent optional stopping are achievable in restless bandits with cointegration.

We describe algorithms for use by prediction markets in forming a crowd consensus joint probability distribution over thousands of related events. Equivalently, we describe market mechanisms to efficiently crowdsource both structure and parameters of a Bayesian network. Prediction markets are among the most accurate methods to combine forecasts; forecasters form a consensus probability distribution by trading contingent securities. A combinatorial prediction market forms a consensus joint distribution over many related events by allowing conditional trades or trades on Boolean combinations of events. Explicitly representing the joint distribution is infeasible, but standard inference algorithms for graphical probability models render it tractable for large numbers of base events. We show how to adapt these algorithms to compute expected assets conditional on a prospective trade, and to find the conditional state where a trader has minimum assets, allowing full asset reuse. We compare the performance of three algorithms: the straightforward algorithm from the DAGGRE (Decomposition-Based Aggregation) prediction market for geopolitical events, the simple block-merge model from the SciCast market for science and technology forecasting, and a more sophisticated algorithm we developed for future markets.

Smart manufacturing relies on a combination of different sources providing key information to support diverse activities throughout the manufacturing process. Most smart manufacturing systems focus on activities directly related to the management of robots, conveyor belts, maintenance logs, and others that ensure the process runs smoothly. An initial step to support such smart manufacturing systems is an awareness process for estimating current situations and predicting future situations in manufacturing, called Predictive Manufacturing Situation Awareness (MSAW). Our research addresses developing an MSAW system with the goal of enhancing industrial competitiveness (e.g., lower cost in shorter time with higher quality) for the manufacturing industry. This requires constant monitoring of market conditions, prices of manufacturing assets, and other inputs that would help to define how the production line behaves. This input is highly stochastic, which makes fusing the data from the diverse sources a challenge. In such situations, the MSAW system requires efficient knowledge representation for various situations and expeditious reasoning methods for estimating current situations as well as predicting future situations. In this paper, we provide an overview of the data fusion process supporting MSAW, including the representation of situations with associated uncertainty, and reasoning methods to support improved manufacturing processes.

The International Society of Information Fusion (ISIF) Evaluation Techniques for Uncertainty Representation Working Group (ETURWG) investigates the quantification and evaluation of all types of uncertainty regarding the inputs, reasoning and outputs of the information fusion process. The ETURWG is developing an Uncertainty Representation and Reasoning Framework (URREF) ontology for this purpose. This paper outlines a start towards the process of defining metrics for the URREF data criteria, which will align the URREF ontology with practical application. A criterion can be evaluated according to several metrics, and a metric can be applied to several criteria. As such, the ontology would have to reflect the nature of a many-to-many mapping between criteria and metrics. The main findings and suggestions of the paper advancing the use of URREF are: 1) The Weight of Information (WoI) is dependent on data criteria, which in turn depend on source criteria. 2) Criteria and metrics that apply to evidence (typically an input of the fusion system), could equally apply to the fusion system outputs or internal information, which in turn could form the inputs of another system. As such the word “Evidence” in the terms “Piece of Evidence” and “Weight of Evidence” should be replaced by the word “Information”. 3) Accuracy and precision and associated metrics are ubiquitous in the URREF ontology and can evaluate many parts of the fusion system. 4) The weight of information also assumes an important position in the ontology, as it depends on several source and data criteria.

The question addressed in this paper is “what” is to be evaluated by the Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology. We thus identify the elements composing uncertainty representation and reasoning approaches, which constitute various subjects being assessed. We distinguish between primary evaluation subjects (Uncertainty Representation and Reasoning components of the fusion algorithm), and secondary evaluation subjects (source of information, piece of information, fusion method and mathematical model). This paper proposes a list of source quality criteria to be added to the ontology and establishes formal links between the secondary and primary evaluation subjects. The key contribution of the paper is the update of the definitions of sub-criteria of the Expressiveness criterion together with suggestions for complementary concepts to be included in the ontology (type of scale, type of uncertainty expression). Conclusions are drawn to extend the work in using the expressiveness criterion for information fusion analysis.

The Profile Hidden Markov Model (PHMM) is commonly used to represent biological sequences. We present a method for transforming the Profile HMM into an equivalent standard HMM where each transition is associated with a single emission. Using this transformation, we develop a beam method, which includes a novel variational adaptation of the infinite-HMM beam sampling technique, to create a fast inference algorithm. We evaluate our algorithm on both synthetic data and protein sequence datasets, showing that our beam method can lead to considerable improvements in runtime while maintaining the model's ability to concisely represent sequences.

A nonparametric Bayesian approach to co-clustering ensembles is presented. Similar to clustering ensembles, co-clustering ensembles combine various base co-clustering results to obtain a more robust consensus co-clustering. To avoid pre-specifying the number of co-clusters, we specify independent Dirichlet process priors for the row and column clusters. Thus, the numbers of row- and column-clusters are unbounded a priori; the actual numbers of clusters can be learned a posteriori from observations. Next, to model non-independence of row- and column-clusters, we employ a Mondrian Process as a prior distribution over partitions of the data matrix. As a result, the co-clusters are not restricted to a regular grid partition, but form nested partitions with varying resolutions. The empirical evaluation demonstrates the effectiveness of nonparametric Bayesian co-clustering ensembles and their advantages over traditional co-clustering methods.

In a net-centric world, systems will be required to fuse data from geographically dispersed, heterogeneous information sources operating asynchronously, to produce up-to-date, mission-relevant knowledge to inform commanders. Realizing this vision requires overcoming a number of technical challenges. Among these is the need for semantic interoperability among systems with different internal data models and vocabularies. Ontologies are seen as a key enabling technology for semantic interoperability. Although information fusion by nature involves reasoning under uncertainty, traditional ontology formalisms provide no principled means of reasoning under uncertainty. This paper proposes the use of probabilistic ontologies within a service-oriented architecture as a means to enable semantic interoperability in net-centric fusion systems.

This paper presents a technical approach for fusing information from diverse sources. Fusion requires appropriate weighting of information based on the quality of the source of the information. A credibility model characterizes the quality of information based on the source and the circumstances under which the information is collected. In many cases credibility is uncertain, so inference is necessary. Explicit probabilistic credibility models provide a computational model of the quality of the information that allows use of prior information, evidence when available, and opportunities for learning from data. This paper provides an overview of the challenges, describes the advanced probabilistic reasoning tools used to implement credibility models, and provides an example of the use of credibility models in a multi-source fusion process

Recent advances in the technology of multiunit recordings make it possible to test Hebb's hypothesis that neurons do not function in isolation but are organized in assemblies. This has created the need for statistical approaches to detecting the presence of spatiotemporal patterns of more than two neurons in neuron spike train data. We mention three possible measures for the presence of higher-order patterns of neural activation--coefficients of log-linear models, connected cumulants, and redundancies--and present arguments in favor of the coefficients of log-linear models. We present test statistics for detecting the presence of higher-order interactions in spike train data by parameterizing these interactions in terms of coefficients of log-linear models. We also present a Bayesian approach for inferring the existence or absence of interactions and estimating their strength. The two methods, the frequentist and the Bayesian one, are shown to be consistent in the sense that interactions that are detected by either method also tend to be detected by the other. A heuristic for the analysis of temporal patterns is also proposed. Finally, a Bayesian test is presented that establishes stochastic differences between recorded segments of data. The methods are applied to experimental data and synthetic data drawn from our statistical models. Our experimental data are drawn from multiunit recordings in the prefrontal cortex of behaving monkeys, the somatosensory cortex of anesthetized rats, and multiunit recordings in the visual cortex of behaving monkeys.

The construction of a large, complex belief network model, like any major system development effort, requires a structured process to manage system design and development. This paper describes a belief network engineering process based on the spiral system lifecycle model. The problem of specifying numerical probability distributions for random variables in a belief network is best treated not in isolation, but within the broader context of the system development effort as a whole. Because structural assumptions determine which numerical probabilities or parameter values need to be specified, there is an interaction between specification of structure and parameters. Evaluation of successive prototypes serves to refine system requirements, ensure that modeling and elicitation effort are focused productively, and prioritize directions of enhancement and improvement for future prototypes. Explicit representation of semantic information associated with probability assessments facilitates tracing of the rationale for modeling decisions, as well as supporting maintenance and enhancement of the knowledge base.

A model is a representation of a system that can be used to answer questions about the system. In many situations in which models are used, there exists no set of universally accepted modeling assumptions. The term model uncertainty commonly refers to uncertainty about a model's structure, as distinguished from uncertainty about parameters. This paper presents alternative formal approaches to treating model uncertainty, discusses methods for using data to reduce model uncertainty, presents approaches for diagnosing inadequate models, and discusses appropriate use of models that are subject to model uncertainty.

The specification of appropriate procedures for making inferences in the context of uncertain, incomplete or imprecise information is a key research issue in a variety of disciplines. Within and among these disciplines, great debates have occurred about the meaning and appropriateness of various proposals for reasoning under uncertainty. At the core of many of these debates are disagreements as to the appropriate meaning of the term "uncertainty", the related term "probability", and what it means to be unsure of one's own uncertainties. This latter issue, uncertainty about one's uncertainty, is commonly referred to as higher order uncertainty. This paper provides a brief overview of various perspectives in higher order uncertainty.

When eliciting a probability model from experts, knowledge engineers may compare the results of the model with expert judgment on test scenarios, then adjust model parameters to bring the behavior of the model more in line with the experts intuition. This paper presents a methodology for analytic computation of sensitivity values in Bayesian network models. Sensitivity values are partial derivatives of output probabilities with respect to parameters being varied in the sensitivity analysis. They measure the impact of small changes in a network parameter on a target probability value or distribution. Sensitivity values can be used to focus knowledge elicitation effort on those parameters having the most impact on outputs of concern. Analytic sensitivity values are computed for an example and compared to sensitivity analysis by direct variation of parameters.>

Practical decision theoretic reasoning requires the construction of local, problem-specific models in which attention is confined to a restricted universe of propositions. Such a restricted universe is called a small world. Managing the construction and revision of small world models can itself be viewed as a meta-level decision problem. This paper presents a theoretical framework for understanding and analyzing many of the issues associated with the management of small world models.>

A reasoning system and associated assessment methodology built on a natural schema for an evidential argument are discussed. This argument schema is based on the underlying causal chains linking conclusions and evidence. The framework couples a probabilistic calculus with qualitative approaches to evidential reasoning. The resulting knowledge structure leads to a natural assessment methodology in which the expert first specifies a qualitative argument from evidence to conclusion. Next the expert specifies a series of premises on which the argument is based. Invalidating any of these premises would disrupt the causal link between evidence and conclusion. The final step is the assessment of the strength of the argument, in the form of degrees of belief for the premises underlying the argument. The expert may also explicitly adopt assumptions affecting the strength of evidential arguments. A higher-level 'metareasoning' process is described, in which assumptions underlying the strength and direction of evidential arguments may be revised in response to conflict.>