Mission Impact Assessment (MIA) is critical for enhancing system effectiveness and ensuring mission success. This article presents Interdependent Mission Impact Assessment ( iMIA ), an interdependent MIA framework that models relationships among mission components and enables probabilistic reasoning under uncertainty. Designed for AI-driven mission systems operating in dynamic, low-data, or poorly observable environments, iMIA addresses the limitations of traditional methods that often rely on overly confident assumptions about adversary behavior. While conventional Hypergame Theory (HGT) captures perceptual uncertainty from asymmetric or inaccurate views, it overlooks epistemic uncertainty arising from limited knowledge. To bridge this gap, we introduce a hybrid Subjective Logic (SL)-based HGT model (SLHG), integrating SL to represent epistemic uncertainty and HGT to account for misperceptions. This integration supports informed decision-making under both uncertain strategy beliefs and divergent environmental views. iMIA evaluates mission impact using multidimensional system quality metrics, security, trust, resilience, and agility, across diverse attacker–defender interactions. It identifies critical nodes influencing mission outcomes and quantifies performance gains from asset capacity reinforcement and asset vulnerability mitigation. Applied to a vehicle-assisted AI-based mission system, iMIA with SLHG improves performance by 16% in \(ASR\) , 20% in \(MTBF\) , 11% in \(TSA\) , and 14% in \(P_{ACC}\) . Designed for incremental development, iMIA supports continuous feedback and iterative refinement. Our results show that feedback-driven adjustments improve overall system performance by up to 18% in the accuracy performance.

An accurate assessment of a mission system’s performance, called mission impact assessment (MIA), can effectively identify and counteract any issues in the current system to avoid potential risks or vulnerabilities that may cause mission failure. Although the importance of MIA research has been recognized to mitigate the system’s potential risk for more than a decade, little research has shown a comprehensive MIA framework with experimental validation showing the inference performance of the MIA tools under realistic attack–defense interactions. To fill this gap, we propose an interdependent MIA, callediMIA, that can adequately capture the interrelationships of the key components in a mission system and environment and accurately infer a mission outcome (i.e., success or failure) under uncertainty. In the proposediMIA, we first consider Subjective Bayesian Networks to evaluate how successfully a given system introduces impacts on mission success under uncertainty due to a lack of evidence. In addition, we consider strategic attack-defense interactions based on a hypergame theory in which the attacker and defender can take actions under their perceived uncertainty. Our extensive experiments showed the outperformance of our proposediMIAup to 70% and 25% over the performance of the baseline and the state-of-the-art Bayesian network (BN)-based counterparts, respectively, in terms of inference accuracy in mission performance (e.g., mission outcomes, such as mission success or failure). We also provide in-depth sensitivity analyses to identify the key system components that should be considered more carefully to avoid mission failure.

Subjective Bayesian networks (SBN) integrate Bayesian Networks (BN) with Subjective Logic, enabling the representation of second-order uncertainty, denoting the uncertainty surrounding the probability distribution of an event. Although prior research predominantly centers on exact inference within the SBN framework, there is a notable dearth of exploration into the realm of approximate inference in SBN. Our work is specifically geared towards addressing this gap, focusing on the application of diverse sampling methodologies (i.e., forward and Gibbs sampling) for approximate inference in SBN. The primary contribution of this work lies not only in the introduction of approximate inference in SBN but also in the formulation of an “anytime” SBN inference algorithm. This implies that a best inference estimate can be obtained at any given moment, given trade-offs in the precision. Moreover, the allocation of computational resources is a customizable and potentially optimizable process. Through a rigorous series of experiments, we empirically demonstrate that the number of iterations to convergence decreases as we provide more samples for both forward and Gibbs sampling. Furthermore, we discover the difference between approximate and exact inference in belief ($\delta_{\text {belief }}$) and uncertainty ($\delta_{\text {uncertainty }}$) mass of subjective opinion becomes more unpredictable as the error gets large in BN probability. Lastly, in our experiments, we demonstrate the number of BN samples has a greater impact on $\delta_{\text {belief }}$ than the number of SBN iterations. These findings indicate that the family of greedy algorithms (based on local graded changes - such as gradients) can be a promising approach for finding optimal allocations of computational resources in this framework. The software assets produced and used in this work will be made available as an open source Python library.

A mission impact assessment (MIA) framework assesses a mission system’s performance and/or aims to identify risk factors of mission failure to take a mitigation strategy. The iMIA framework, unlike traditional MIA approaches, comprehensively addresses the interdependencies of key system components like asset vulnerabilities, attack behaviors, defense mechanisms, and service/task characteristics. This framework goes beyond conceptual models, providing a validated and detailed MIA framework covering from a conceptual model to detailed mission and system designs. Existing MIA approaches with Bayesian Networks (BNs) overlook inherent uncertainties arising from factors like insufficient evidence or data in real-world applications. To address decision-making challenges in the face of uncertainties, we introduce Subjective Bayesian Networks (SBNs). SBNs estimate uncertainties and interpret them using an expert’s domain knowledge or historical data, forming a subjective opinion through prior belief in SBNs. Using the SBNs, we enhance iMIA’s inference accuracy in a Federated Learning-based mission system (FLMS) for vehicular networks. Our experiments show thatiMIA’s SBN reasoning significantly improves mission outcome inference accuracy compared to conventional BN-based reasoning in existing MIA approaches. We also evaluate mission performance based on service availability and prediction accuracy from the FLMS.

In this work, we present the first dataset, MAILEX, for performing event extraction from conversational email threads.To this end, we first proposed a new taxonomy covering 10 event types and 76 arguments in the email domain.Our final dataset includes 1.5K email threads and ∼4K emails, which are annotated with totally ∼8K event instances.To understand the task challenges, we conducted a series of experiments comparing three types of approaches, i.e., fine-tuned sequence labeling, fine-tuned generative extraction, and few-shot in-context learning.Our results showed that the task of email event extraction is far from being addressed, due to challenges lying in, e.g., extracting non-continuous, shared trigger spans, extracting non-named entity arguments, and modeling the email conversational history.Our work thus suggests more future investigations in this domain-specific event extraction task. 1

Test and Evaluation for verification and validation (V&V) of sensor data fusion techniques utilize methods of uncertainty analysis. The Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology identifies many attributes of metrics (i.e., semantic meaning, object metrics, and subjective quality). With the growing interest in artificial intelligence (AI) due to large data corpus access, fast compute power, and machine/deep learning (ML/DL) techniques; V&V of these methods are needed. In this paper, the enhancement of the URREF to utilize a risk assessment for decision is demonstrated towards analysis/alignment of ML/DL methods that utilize multi-modal data fusion. Evidential reasoning is considered in the use case to provide data handing reliability source and processing credibility to measure decision risk in a maritime domain awareness scenario.

Research in Natural Language Processing, bio-medicine, and computer vision achieved excellent results in machine learning due to the success of the Transformer-based models. However, these excellent results depend on the labeled high-quality and large-scale datasets. If one of these requirements is not met, the model may lack generalization ability, and its performance will be unsatisfactory. To address these issues, this research proposes a Generic Multimodal Gradient-Based Meta Framework (GeMGF) trained from scratch to avoid language bias, learns from a few data, and reduces the model degradation trained on a finite dataset. GeMGF was evaluated using the benchmark dataset CUB-200-2011 for the text and image classification tasks. The results show that GeMGF outperforms the state-of-the-art models with 93.2% accuracy. GeMGF is simple, efficient, and adaptable to other data modalities and fields.

ChatGPT, OpenAI’s chatbot, has gained consider-able attention since its launch in November 2022, owing to its ability to formulate articulated responses to text queries and comments relating to seemingly any conceivable subject. As impressive as the majority of interactions with ChatGPT are, this large language model has a number of acknowledged shortcomings, which in several cases, may be directly related to how ChatGPT handles uncertainty. The objective of this paper is to pave the way to formal analysis of ChatGPT uncertainty handling. To this end, the ability of the Uncertainty Representation and Reasoning Framework (URREF) ontology is assessed, to support such analysis. Elements of structured experiments for reproducible results are identified. The dataset built varies Information Criteria of Correctness, Non-specificity, Self-confidence, Relevance and Inconsistency, and the Source Criteria of Reliability, Competency and Type. ChatGPT’s answers are analyzed along Information Criteria of Correctness, Non-specificity and Self-confidence. Both generic and singular information are sequentially provided. The outcome of this preliminary study is twofold: Firstly, we validate that the experimental setup is efficient in capturing aspects of ChatGPT uncertainty handling. Secondly, we identify possible modifications to the URREF ontology that will be discussed and eventually implemented in URREF ontology Version 4.0 under development.

Subjective Bayesian networks (SBN) combine Bayesian Networks (BN) with Subjective Logic in order to express the second-order uncertainty (i.e., the uncertainty about a probability distribution of an event – as opposed to the uncertainty about the event itself). While SBNs provide a strong formalism for treating the uncertainty in a higher level, the literature lacks support for extensive software implementations focused on compatibility with current software solutions or standards. Our work explores the structural congruence between BN and SBN (in terms of software data structure) and a semantic bijection between Subjective Logic opinions to Dirichlet distributions to introduce a SBN reasoning framework that targets on effectively reusing the existing BN solutions. Particularly, we developed two inference algorithms that apply a Monte Carlo method to existing BN inference algorithms (we chose the Junction Tree algorithm, for test), respectively for batch and interactive SBN reasoning. A method for translating an evidence in SBN to uncertain (virtual) evidence in BN is also presented. The main contribution of this paper is the introduction of a simple, yet flexible empirical estimation method and a software architecture that virtually adapts any BN inference algorithm to an approximate computational inference framework for SBN. We also developed a Java component to demonstrate the reusability, and we developed a case study of Mission Impact Assessment of Unmanned Aerial Vehicles transporting critical items between hospitals in order to illustrate the applicability in a knowledge engineering process.

Advanced Persistent Threats (APTs) are extremely dangerous and hidden cyber threats. Even with the unavailability of computing assets and data penetration using cloud computing, APTs still find ways to exploit vulnerabilities. This paper proposes new methods for classifying malicious traffic. Since cyber attacks have not yet been widely studied, they are similar to Human Activity Recognition (HAR). In this sense, we introduce the Transformer, an effective Machine Learning approach, with a new architecture that shows more remarkable effects in the classification of HAR. In preliminary experiments using our model, the classification accuracy reached 91.45%, and the average accuracy reached 91.25 %. We also applied other classification algorithms, such as RNN, CNN, and LSTM, which also presented adequate solutions for the described cyber attack.

Mission Impact Assessment (MIA) is a critical endeavor for evaluating the performance of mission systems, encompassing intricate elements such as assets, services, tasks, vulnerability, attacks, and defenses. This study introduces an innovative MIA framework that transcends existing methodologies by intricately modeling the interdependencies among these components. Additionally, we integrate hypergame theory to address the strategic dynamics of attack-defense interactions. To illustrate its practicality, we apply the framework to an Internet-of-Things (IoT)-based mission system tailored for accurate, time-sensitive object detection. Rigorous simulation experiments affirm the framework's robustness across a spectrum of scenarios. Our results prove that the developed MIA framework shows a sufficiently high inference accuracy (e.g., 80 %) even with a small portion of the training dataset (e.g., 20–50 %).

This paper explores the use of the Uncertainty Representation and Reasoning Evaluation Framework (URREF), a framework intended to change that state of affairs, in evaluating potential uncertainty representation approaches for a maritime Decision Support System. We revisit some comparison aspects discussed along the years and map them to the URREF. We illustrate the comparison on a simple maritime use case involving basic reasoning about threat assessment, with observations from partially reliable sources. The same fusion problem is modeled with the two uncertainty theories of Bayesian probability theory and evidence theory. Within the same framework, we consider two different reasoning schemes, Causal and Evidential, complemented with a source model of partial reliability. Comparison items are mapped to URREF ontology criteria of (Representation) Expressiveness and (Reasoning) Correctness. We highlight the criteria that can be useful in supporting systems developers in their choice of how to represent and manage uncertainty in information fusion processes, and propose some refinement to the URREF to capture handling of inconsistency.

In this paper we evaluate the performance of question generation using contextual reasoning in dialogs generated from an Ontology to provide identification, authentication, and access control. Generating relevant questions requires frequent querying after each question, using its context to generate the next one. This process is tedious and might lead to a situation where the interviewee has to wait for a question for an unacceptably long time. The objective of this work is to show that our method for generation of questions using context does not cause delays that might turn the dialogs useless.

High-Level Information Fusion (HLIF) provides the ability to combine data from diverse sources, including documents involving analyst assessment and raw sensor reports generated by sensors, in a coherent and consistent way. Command and Control (C2) in cyber infrastructure involves gathering information from experts, merging it with field knowledge and experimental results, and selected the most appropriate cyber assets to deploy at any given time in the mission cycle. When framing cyber asset selection as a HLIF problem, one key aspect involves estimation of network-wide impacts generated by cyber assets. Cyberspace is a highly dynamic man-made domain with a high degree of uncertainty and incomplete data which must be transformed into knowledge to support precise and predictable cyber effects estimation. Current systems have to rely on human subject matter experts (SMEs) for most tasks, rendering the cyber asset planning process too time consuming and therefore operationally ineffective. This paper proposes an architecture that leverages probabilistic ontologies to expedite the cyber asset planning process, allowing for the automation of most time-consuming, error-prone, SME-based knowledge elicitation under uncertainty. We illustrate the main aspects of the proposed architecture through examples taken from the Derived and Integrated Cyber Assets (DICE) project.

The choices of the uncertainty representations and reasoning methods have a critical impact on the development and deployment of modern fusion solutions. They influence the development effort, the quality of the resulting solutions as well as the deployment costs. However, such choices require an analysis that considers many operational and theoretical aspects. The Uncertainty Representation and Reasoning Evaluation Framework (URREF) concepts enable such an analysis. The proposed URREF -driven development approach establishes relations between the URREF criteria and evaluation subjects in the context of a development and deployment life cycle. In this way the relevant theoretical elements of the fusion techniques employed are emphasized and evaluated at various stages of the development process, facilitating informed design choices as well as systematic and tractable evaluation of complex fusion solutions. The concepts are illustrated through the assessment of a high-level fusion approach supporting estimation of the whereabouts of wildlife poachers.

Aiding decision-makers is a key function of a fusion system. In designing decision-aiding modules for fusion systems, it is necessary to understand the elements of the decision model and the dependencies that connect them. An ontology is a disciplined means to codify that understanding. Many fusion systems have a Bayesian Network (BN) component to support probabilistic reasoning under uncertainty. Decision graphs (DG) are an extension that adds decision aiding to BNs. Both BNs and DGs have limited logical expressivity, able to model propositions, but cannot directly model variable numbers of entities or variations in their attributes and relationships. This important capability is called first-order expressivity. Multi-Entity Bayesian Network (MEBN) was developed to provide first-order logic expressivity to BNs. We are developing Multi-Entity Decision Graph (MEDG) to do the same for decision graphs. We found that a decision ontology is useful to our efforts. The literature has a limited discussion of decision ontologies. Almost all focus on the entities and the entity hierarchy. But BNs and DGs emphasize relationships and the dependencies between relationships. The key for probabilistic first-order expressivity is to identify the relationships that enable dependencies between entity instances. We developed a MEDG Decision Ontology that highlights both the entities and key relationships that any decision model needs to address. It is designed to support decision model developers, including fusion model developers, in building comprehensive decision aiding capabilities.

The International Society of Information Fusion (ISIF) Evaluation Techniques for Uncertainty Representation Working Group (ETURWG) investigates the quantification and evaluation of all types of uncertainty regarding the inputs, reasoning and outputs of the information fusion process. The ETURWG is developing an Uncertainty Representation and Reasoning Framework (URREF) ontology for this purpose. This paper outlines a start towards the process of defining metrics for the URREF data criteria, which will align the URREF ontology with practical application. A criterion can be evaluated according to several metrics, and a metric can be applied to several criteria. As such, the ontology would have to reflect the nature of a many-to-many mapping between criteria and metrics. The main findings and suggestions of the paper advancing the use of URREF are: 1) The Weight of Information (WoI) is dependent on data criteria, which in turn depend on source criteria. 2) Criteria and metrics that apply to evidence (typically an input of the fusion system), could equally apply to the fusion system outputs or internal information, which in turn could form the inputs of another system. As such the word “Evidence” in the terms “Piece of Evidence” and “Weight of Evidence” should be replaced by the word “Information”. 3) Accuracy and precision and associated metrics are ubiquitous in the URREF ontology and can evaluate many parts of the fusion system. 4) The weight of information also assumes an important position in the ontology, as it depends on several source and data criteria.

The question addressed in this paper is “what” is to be evaluated by the Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology. We thus identify the elements composing uncertainty representation and reasoning approaches, which constitute various subjects being assessed. We distinguish between primary evaluation subjects (Uncertainty Representation and Reasoning components of the fusion algorithm), and secondary evaluation subjects (source of information, piece of information, fusion method and mathematical model). This paper proposes a list of source quality criteria to be added to the ontology and establishes formal links between the secondary and primary evaluation subjects. The key contribution of the paper is the update of the definitions of sub-criteria of the Expressiveness criterion together with suggestions for complementary concepts to be included in the ontology (type of scale, type of uncertainty expression). Conclusions are drawn to extend the work in using the expressiveness criterion for information fusion analysis.

Planning courses of action for a task force component involves analyzing a considerable amount of data, as well as assessing the available options and their respective impact to the operational goals. This is a time consuming process that is dramatically restricted by the currently established doctrine, which allocates a too short period for this task during its decision cycle. An inefficient COA planning generates losses of resources and compromises the ability of a force to achieve the desired effects in each mission. This study investigates the possibilities of using semantics to support courses of action modeling and simulation via a web services architecture. Our preliminary results indicate that with this semantics-based approach it is possible to automatically generate more than one course of action analysis on the basis of the commander´s intent in a timely manner.

In a net-centric world, systems will be required to fuse data from geographically dispersed, heterogeneous information sources operating asynchronously, to produce up-to-date, mission-relevant knowledge to inform commanders. Realizing this vision requires overcoming a number of technical challenges. Among these is the need for semantic interoperability among systems with different internal data models and vocabularies. Ontologies are seen as a key enabling technology for semantic interoperability. Although information fusion by nature involves reasoning under uncertainty, traditional ontology formalisms provide no principled means of reasoning under uncertainty. This paper proposes the use of probabilistic ontologies within a service-oriented architecture as a means to enable semantic interoperability in net-centric fusion systems.

As the work with semantics and services grows more ambitious in the semantic Web community, there is an increasing appreciation on the need for principled approaches for representing and reasoning under uncertainty. Reacting to this trend, the World Wide Web Consortium (W3C) has created the Uncertainty Reasoning for the World Wide Web Incubator Group (URW3-XG) to better define the challenge of reasoning with and representing uncertain information available through the World Wide Web and related WWW technologies. In according to the URW3-XG effort this paper presents the implementation of a graphical user interface for building probabilistic ontologies, an application programming interface for saving and loading these ontologies and a proposal to specify formulas for creating conditional probabilistic tables dynamically. The language used for building probabilistic ontologies is probabilistic OWL (Pr-OWL), an extension for OWL based on multi-entity Bayesian network (MEBN).