Advancements in conversational artificial intelligence (AI) enable scalable, web-based training environments beyond traditional classroom settings to support learning. In emergency communication centers such as 9-1-1, trainees have limited opportunities to practice diverse call-handling scenarios, as it relies on role-play instruction and constrained instructor availability. We introduce SHIELD (Strengthening Human Intervention in Emergencies through Learning with Data and AI), a framework and online conversational AI system designed to support scenario-based training through interactive simulations, adaptive feedback, and learning analytics. SHIELD integrates AI-generated call simulations with data science techniques to capture fine-grained trainee interaction data, including decision-making behavior, response timing, and corrective actions. These logged interactions are analyzed to provide real-time metacognitive nudges during simulated calls and post-performance feedback through AI-assisted analytics. The framework is informed by principles of self-regulated learning to structure training across planning, execution, and reflection phases. This demo showcases SHIELD as a web-based prototype deployed for emergency call-taker training and discusses design insights from initial testing sessions conducted in collaboration with a public safety communications agency. The system also highlights potential applicability to other high-stakes operational training settings.

When employing neural networks in the real world, we often encounter challenges related to dataset size, data imbalance and model selection. In this paper, we investigate the benefits of incorporating information obtained from causal structures or Qualitative Models for Data Generation Processes (QM-DGP) into neural network design and training, by leveraging causal reasoning principles such as d-separation and the Markov blanket to determine relevant input variables. Through empirical analysis, we compare networks which exploit causal structures and those that do not, focusing on aspects related to dataset efficiency. Our findings show that networks which exploit causal structures require fewer samples to achieve (near-)optimal performance on average, making them more suitable in frugal learning scenarios. We also propose a causally-decomposed neural network architecture based on causal structural information and show that it is more data efficient than its fully-connected counterpart. Our results highlight the practical advantage of using causal reasoning in neural network design, particularly in settings where data efficiency plays a crucial role.

In today's digital landscape, phishing attacks persist as a formidable challenge, highlighting the need for robust strategies to mitigate individual risk. While advanced machine learning techniques have excelled in identifying those most susceptible to phishing, existing research has primarily focused on refining prediction accuracy rather than leveraging this understanding to mitigate risk. To bridge this gap, we present a novel counterfactual explanation approach aimed at identifying the specific traits that heighten an individual's vulnerability to phishing. Our approach integrates uncertainties and causal insights from the data generation process, producing actionable intelligence to effectively lower individual susceptibility. This enables us to tailor personalized recommendations to reduce individual's vulnerability. Through experimentation, we assess the efficacy of our methodology and demonstrate capacity to reduce susceptibility to phishing. These findings emphasize the importance of personalized interventions, arming individuals with the knowledge necessary to improve their online security protocols.

This paper proposes an improved analysis of the Qualitative Models of Data Generating Processes (QM-DGP). The approach supports (i) determination of the complexity of a Machine Learning problem and (ii) a coarse determination of the quantities of training data that are needed to train good quality models. Compared to the previously published approach to the QM-DGP analysis, this paper introduces a more thorough and theoretically sound treatment of the learning complexity. Firstly, the approach provides more rigorous determination of the complexity of the data generating processes (DGP). Secondly, the determination of the learning complexity and the required training data volumes is based on sound statistical principles for the estimation of the distributions over categorical variables. The effectiveness of the proposed method was experimentally confirmed in controlled settings. Different ground truth models were used to sample test and training data. The approach correctly predicts the size of the training data sets for which machine learning yields models supporting classification close to Bayes Error. While the majority of the experiments were carried out on probabilistic graphical models (PGM), the experiments with Neural Networks confirmed that the QM-DGP approach is not limited to PGMs.

Test and Evaluation for verification and validation (V&V) of sensor data fusion techniques utilize methods of uncertainty analysis. The Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology identifies many attributes of metrics (i.e., semantic meaning, object metrics, and subjective quality). With the growing interest in artificial intelligence (AI) due to large data corpus access, fast compute power, and machine/deep learning (ML/DL) techniques; V&V of these methods are needed. In this paper, the enhancement of the URREF to utilize a risk assessment for decision is demonstrated towards analysis/alignment of ML/DL methods that utilize multi-modal data fusion. Evidential reasoning is considered in the use case to provide data handing reliability source and processing credibility to measure decision risk in a maritime domain awareness scenario.

ChatGPT, OpenAI’s chatbot, has gained consider-able attention since its launch in November 2022, owing to its ability to formulate articulated responses to text queries and comments relating to seemingly any conceivable subject. As impressive as the majority of interactions with ChatGPT are, this large language model has a number of acknowledged shortcomings, which in several cases, may be directly related to how ChatGPT handles uncertainty. The objective of this paper is to pave the way to formal analysis of ChatGPT uncertainty handling. To this end, the ability of the Uncertainty Representation and Reasoning Framework (URREF) ontology is assessed, to support such analysis. Elements of structured experiments for reproducible results are identified. The dataset built varies Information Criteria of Correctness, Non-specificity, Self-confidence, Relevance and Inconsistency, and the Source Criteria of Reliability, Competency and Type. ChatGPT’s answers are analyzed along Information Criteria of Correctness, Non-specificity and Self-confidence. Both generic and singular information are sequentially provided. The outcome of this preliminary study is twofold: Firstly, we validate that the experimental setup is efficient in capturing aspects of ChatGPT uncertainty handling. Secondly, we identify possible modifications to the URREF ontology that will be discussed and eventually implemented in URREF ontology Version 4.0 under development.

AI-based decision support solutions require life cycles that adequately address critical steps, such as (i) finding suitable machine learning (ML) methods for the problem at hand, (ii) preparing and executing adequate data acquisition processes and (iii) tractable evaluation of the overall solution. Understanding the data generating processes is key in achieving this. Training and test data can be seen as a result of a causal data generation process, a sampling process in which the data is collected from different sources that are influenced by multiple interdependent phenomena. This is represented by a Qualitative Model of Data Generation Processes (QM-DGP), a causal graphical model. QM-DGP facilitates analysis of the complexity of the underlying data generating processes that can inform the development of trustable ML-based solutions in multiple ways. Firstly, this analysis is the basis for the determination of the required complexity of the ML models. Secondly, it facilitates the determination of the quantities of training data supporting good learning results. Thirdly, it can provide guidance for a systematic simplification of the models, supporting tractable solutions without significantly reduced performance. The construction of QM-DGP and the analysis benefit from sound theoretical concepts, such as d-separation and I-Maps. Experimental results with simulated data indicate that the approach can be effective in predicting the required quantities of training data and the determination of the modelling complexity using different types of models.

Bayesian bandits using Thompson Sampling have seen increasing success in recent years. Yet existing value models (of rewards) are misspecified on many real-world problem. We demonstrate this on the User Experience Optimization (UXO) problem, providing a novel formulation as a restless, sleeping bandit with unobserved confounders plus optional stopping. Our case studies show how common misspecifications can lead to sub-optimal rewards, and we provide model extensions to address these, along with a scientific model building process practitioners can adopt or adapt to solve their own unique problems. To our knowledge, this is the first study showing the effects of overdispersion on bandit explore/exploit efficacy, tying the common notions of under- and over-confidence to over- and under-exploration, respectively. We also present the first model to demonstrate that vanishing regret and fast and consistent optional stopping are achievable in restless bandits with cointegration.

Smart manufacturing relies on a combination of different sources providing key information to support diverse activities throughout the manufacturing process. Most smart manufacturing systems focus on activities directly related to the management of robots, conveyor belts, maintenance logs, and others that ensure the process runs smoothly. An initial step to support such smart manufacturing systems is an awareness process for estimating current situations and predicting future situations in manufacturing, called Predictive Manufacturing Situation Awareness (MSAW). Our research addresses developing an MSAW system with the goal of enhancing industrial competitiveness (e.g., lower cost in shorter time with higher quality) for the manufacturing industry. This requires constant monitoring of market conditions, prices of manufacturing assets, and other inputs that would help to define how the production line behaves. This input is highly stochastic, which makes fusing the data from the diverse sources a challenge. In such situations, the MSAW system requires efficient knowledge representation for various situations and expeditious reasoning methods for estimating current situations as well as predicting future situations. In this paper, we provide an overview of the data fusion process supporting MSAW, including the representation of situations with associated uncertainty, and reasoning methods to support improved manufacturing processes.

The International Society of Information Fusion (ISIF) Evaluation Techniques for Uncertainty Representation Working Group (ETURWG) investigates the quantification and evaluation of all types of uncertainty regarding the inputs, reasoning and outputs of the information fusion process. The ETURWG is developing an Uncertainty Representation and Reasoning Framework (URREF) ontology for this purpose. This paper outlines a start towards the process of defining metrics for the URREF data criteria, which will align the URREF ontology with practical application. A criterion can be evaluated according to several metrics, and a metric can be applied to several criteria. As such, the ontology would have to reflect the nature of a many-to-many mapping between criteria and metrics. The main findings and suggestions of the paper advancing the use of URREF are: 1) The Weight of Information (WoI) is dependent on data criteria, which in turn depend on source criteria. 2) Criteria and metrics that apply to evidence (typically an input of the fusion system), could equally apply to the fusion system outputs or internal information, which in turn could form the inputs of another system. As such the word “Evidence” in the terms “Piece of Evidence” and “Weight of Evidence” should be replaced by the word “Information”. 3) Accuracy and precision and associated metrics are ubiquitous in the URREF ontology and can evaluate many parts of the fusion system. 4) The weight of information also assumes an important position in the ontology, as it depends on several source and data criteria.

The question addressed in this paper is “what” is to be evaluated by the Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology. We thus identify the elements composing uncertainty representation and reasoning approaches, which constitute various subjects being assessed. We distinguish between primary evaluation subjects (Uncertainty Representation and Reasoning components of the fusion algorithm), and secondary evaluation subjects (source of information, piece of information, fusion method and mathematical model). This paper proposes a list of source quality criteria to be added to the ontology and establishes formal links between the secondary and primary evaluation subjects. The key contribution of the paper is the update of the definitions of sub-criteria of the Expressiveness criterion together with suggestions for complementary concepts to be included in the ontology (type of scale, type of uncertainty expression). Conclusions are drawn to extend the work in using the expressiveness criterion for information fusion analysis.

The Profile Hidden Markov Model (PHMM) is commonly used to represent biological sequences. We present a method for transforming the Profile HMM into an equivalent standard HMM where each transition is associated with a single emission. Using this transformation, we develop a beam method, which includes a novel variational adaptation of the infinite-HMM beam sampling technique, to create a fast inference algorithm. We evaluate our algorithm on both synthetic data and protein sequence datasets, showing that our beam method can lead to considerable improvements in runtime while maintaining the model's ability to concisely represent sequences.

A nonparametric Bayesian approach to co-clustering ensembles is presented. Similar to clustering ensembles, co-clustering ensembles combine various base co-clustering results to obtain a more robust consensus co-clustering. To avoid pre-specifying the number of co-clusters, we specify independent Dirichlet process priors for the row and column clusters. Thus, the numbers of row- and column-clusters are unbounded a priori; the actual numbers of clusters can be learned a posteriori from observations. Next, to model non-independence of row- and column-clusters, we employ a Mondrian Process as a prior distribution over partitions of the data matrix. As a result, the co-clusters are not restricted to a regular grid partition, but form nested partitions with varying resolutions. The empirical evaluation demonstrates the effectiveness of nonparametric Bayesian co-clustering ensembles and their advantages over traditional co-clustering methods.

In a net-centric world, systems will be required to fuse data from geographically dispersed, heterogeneous information sources operating asynchronously, to produce up-to-date, mission-relevant knowledge to inform commanders. Realizing this vision requires overcoming a number of technical challenges. Among these is the need for semantic interoperability among systems with different internal data models and vocabularies. Ontologies are seen as a key enabling technology for semantic interoperability. Although information fusion by nature involves reasoning under uncertainty, traditional ontology formalisms provide no principled means of reasoning under uncertainty. This paper proposes the use of probabilistic ontologies within a service-oriented architecture as a means to enable semantic interoperability in net-centric fusion systems.

This paper presents a technical approach for fusing information from diverse sources. Fusion requires appropriate weighting of information based on the quality of the source of the information. A credibility model characterizes the quality of information based on the source and the circumstances under which the information is collected. In many cases credibility is uncertain, so inference is necessary. Explicit probabilistic credibility models provide a computational model of the quality of the information that allows use of prior information, evidence when available, and opportunities for learning from data. This paper provides an overview of the challenges, describes the advanced probabilistic reasoning tools used to implement credibility models, and provides an example of the use of credibility models in a multi-source fusion process

The construction of a large, complex belief network model, like any major system development effort, requires a structured process to manage system design and development. This paper describes a belief network engineering process based on the spiral system lifecycle model. The problem of specifying numerical probability distributions for random variables in a belief network is best treated not in isolation, but within the broader context of the system development effort as a whole. Because structural assumptions determine which numerical probabilities or parameter values need to be specified, there is an interaction between specification of structure and parameters. Evaluation of successive prototypes serves to refine system requirements, ensure that modeling and elicitation effort are focused productively, and prioritize directions of enhancement and improvement for future prototypes. Explicit representation of semantic information associated with probability assessments facilitates tracing of the rationale for modeling decisions, as well as supporting maintenance and enhancement of the knowledge base.