In the domain of extended object tracking, the tracker's output includes detailed extent information, which complicates the task of accurately measuring the distance between the estimated and actual ground truth. This complexity arises from the need to account for both the spatial extent and the dynamic interactions within groups of objects, such as drone swarms. To address this challenge, this paper introduces a measure designed to evaluate the performance of group tracking algorithms, suited to real world situations. The distance combines location error and group feature errors for a more comprehensive assessment of performances. We illustrate the behavior of the proposed distance on real data collected in an operationally controlled environment. For illustration purposes, we compare two state-of-the-art approaches in their ability to accurately track a drone swarm in a real world situation with bird flights. We also propose a spatial sampling version of the classical OSPA metric as a baseline for comparison. The results show that the proposed distance better captures the trackers' efficiency and enables an easier interpretation of their behavior. We conclude on future work which includes studying properties of the distance in a simulation controlled environment.

In urban drone surveillance, conventional anomaly detection methods often falter amid unstable data and high noise levels. We propose a Sim2Real approach within a ValuationBased System framework that integrates expert knowledge into its model structure. Our method involves two phases: training on simulated drone trajectories to learn behavioral dependencies with probabilities, using a Dynamic Bayesian Network, and performing inference via an evidential network for explicit uncertainty modeling. The incorporation of expert-driven structural insights enhances the transfer of simulation-trained parameters to real-world conditions. Real-world tests with actual flight data demonstrate that our approach outperforms traditional techniques in detection accuracy and recall, robustly handling sensor noise and incomplete information.

Evaluating uncertainty in large language model (LLM) reasoning is challenging due to their vast parameter space, abstract knowledge representation, and limited transparency regarding training data. While normative formalisms, such as deductive logic, clearly define sound reasoning in the absence of uncertainty, reasoning under uncertainty admits multiple approaches, including probabilistic reasoning (e.g. Bayesian), belief function reasoning (e.g. Dempster-Shafer), or fuzzy logic, to name a few. This paper examines how LLMs handle uncertainty by analyzing outcomes based on an atomic fusion and reasoning problem. We establish a point of reference using the simplest of fusion topologies to facilitate transparency and understanding of how LLMs align with established theories. The reasoning approaches of different LLMs with varying complexities are compared to established normative frameworks, providing insights into which formalism best aligns with LLM reasoning and assessing its soundness and consistency. A deviation function for assessment is developed, and the results indicate that the tested LLMs' reasoning under uncertainty does not consistently align with established theories, even for the simplest information fusion topologies. These preliminary results form the basis for further investigations and LLM refinements.

Evaluation of fusion algorithms constitutes a pivotal phase in the life-cycle of designing solutions for fusion problems. Benchmark datasets together with standardized evaluation criteria and metrics are thus needed to run proper experiments. When it comes to higher-level tasks such as situation assessment, datasets are conspicuously absent. Among the reasons are the lack of formal characterization of situation assessment, and the lack of structure to guide the collection of data at an appropriate level of semantics and granularity. In this paper, we propose a model for an imperfect knowledge base (IKB) to support high-level information fusion experiments. The model includes characterization of imperfect information from partially reliable sources. The concept of infon is an elementary piece of information which can bear on both concrete and abstract objects. The corresponding knowledge base encodes infons about physical objects of interest in a situation (e.g., a vessel), about sources providing infons, and about other infons. The imperfection dimensions of information are captured and connected to reliability dimensions of the source. An implementation within a graph database is proposed, exemplified with data of a maritime scenario. An example of comparison of artificial and human agents executing a fusion task is shown. Future work are finally briefly discussed.

Evaluating information fusion algorithms and systems is instrumental to the proper prediction of error, to the rational improvement of solutions and in fine to the acceptance of solutions by end-users. Performance criteria define general semantics for a desirable behavior of the systems, while corresponding metrics implement that semantics for computable quality. Evaluation of the first levels of the JDL model of data fusion (detection and individual object assessment) is classically measured with objective and more or less standardized metrics. Evaluation of higher levels of processing such as situation assessment is less formalized as the evaluation criteria seat somewhere between the tactical picture quality and the decision-maker situation awareness. In this paper, we propose a formalization of the situation assessment problem, which bridges level 1 and 2 of the JDL model. Secondly, we define a global measure of quality encompassing the criteria of completeness, accuracy, clarity, which can be applied to both level 1 and level 2. We illustrate the metrics on a Counter-Unmanned-Aerial System (CUAS) scenario, comparing two uncertainty handling methods for a threat assessment solution through a Dynamic Bayesian Network. We finally conclude and sketch ideas for future steps of this research.

Test and Evaluation for verification and validation (V&V) of sensor data fusion techniques utilize methods of uncertainty analysis. The Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology identifies many attributes of metrics (i.e., semantic meaning, object metrics, and subjective quality). With the growing interest in artificial intelligence (AI) due to large data corpus access, fast compute power, and machine/deep learning (ML/DL) techniques; V&V of these methods are needed. In this paper, the enhancement of the URREF to utilize a risk assessment for decision is demonstrated towards analysis/alignment of ML/DL methods that utilize multi-modal data fusion. Evidential reasoning is considered in the use case to provide data handing reliability source and processing credibility to measure decision risk in a maritime domain awareness scenario.

ChatGPT, OpenAI’s chatbot, has gained consider-able attention since its launch in November 2022, owing to its ability to formulate articulated responses to text queries and comments relating to seemingly any conceivable subject. As impressive as the majority of interactions with ChatGPT are, this large language model has a number of acknowledged shortcomings, which in several cases, may be directly related to how ChatGPT handles uncertainty. The objective of this paper is to pave the way to formal analysis of ChatGPT uncertainty handling. To this end, the ability of the Uncertainty Representation and Reasoning Framework (URREF) ontology is assessed, to support such analysis. Elements of structured experiments for reproducible results are identified. The dataset built varies Information Criteria of Correctness, Non-specificity, Self-confidence, Relevance and Inconsistency, and the Source Criteria of Reliability, Competency and Type. ChatGPT’s answers are analyzed along Information Criteria of Correctness, Non-specificity and Self-confidence. Both generic and singular information are sequentially provided. The outcome of this preliminary study is twofold: Firstly, we validate that the experimental setup is efficient in capturing aspects of ChatGPT uncertainty handling. Secondly, we identify possible modifications to the URREF ontology that will be discussed and eventually implemented in URREF ontology Version 4.0 under development.

Epistemic decisions about which sources to trust and query are critical for a decision-maker, when the end-goal decisions are to be made using limited resources. Toward this, we previously proposed preliminary extensions to classical measures of Value of Information (VoI) for imprecise belief states represented by belief functions relying on a general observation model. These methods primarily aim to assist a decision-maker toward making rational decisions, by generating necessary metadata about the information that is being considered for the decision-making task (e.g. probability of source reliability, degree of self-confidence expressed by the source). In this paper, we explore the behavior and performance of our previously proposed belief theoretic VoI measures, the Evidential Expecetd Value of Sampled Information (EEVSI), and propose a procedure to sequentially select sources to query. We also consider the case where the information sources are providing contradictory evidence. We leverage a maritime surveillance scenario, where the decision-maker has to make a rational ordered selection of information sources among a set of both physical sensors and human sources, to illustrate the behavior of the proposed method. We compare the proposed policy with a traditional probability-based approach in a simulation environment. We conclude by providing some insights on future research directions to further expand on our proposed new measures.

Countering unmanned aerial threats is critical for both military and civilian surveillance and protection systems. The complementarity and redundancy of sensors enables detecting, tracking and classifying Unmanned Aerial Vehicles (UAVs), further assessing their possible threat and planning the proper counter-measure. Despite efforts to develop dedicated systems (sensors, processing, hard and soft kill systems), Counter Unmanned Aerial Systems (CUAS) are often deployed in complex civilian areas, which requires an adequate understanding of the situation including the UAVs behavior for an appropriate response. The evaluation of a CUAS should thus consider not only its ability to provide a tactical picture of sufficient quality, but also its ability to provide semantic information, to establish possible links between objects and to contextualize their behavior. In this paper, we set up the basics for an evaluation platform covering the lower levels but also higher levels of the JDL (Joint Directors of Laboratories) functional model of information fusion. We consider the six classical criteria of a tactical picture (used in Level 1) and extend them to higher level tasks. Evaluation criteria are aligned with the URREF (Uncertainty Representation and Reasoning Evaluation Framework) ontology evaluation criteria. Furthermore, we propose some metrics to quantity such evaluation criteria. The framework is illustrated in a CUAS scenario, with data provided by the LEXLUTOR platform. Several fusion solutions are compared which differ in the way uncertainty is represented, handled and provided to the user.

This paper explores the use of the Uncertainty Representation and Reasoning Evaluation Framework (URREF), a framework intended to change that state of affairs, in evaluating potential uncertainty representation approaches for a maritime Decision Support System. We revisit some comparison aspects discussed along the years and map them to the URREF. We illustrate the comparison on a simple maritime use case involving basic reasoning about threat assessment, with observations from partially reliable sources. The same fusion problem is modeled with the two uncertainty theories of Bayesian probability theory and evidence theory. Within the same framework, we consider two different reasoning schemes, Causal and Evidential, complemented with a source model of partial reliability. Comparison items are mapped to URREF ontology criteria of (Representation) Expressiveness and (Reasoning) Correctness. We highlight the criteria that can be useful in supporting systems developers in their choice of how to represent and manage uncertainty in information fusion processes, and propose some refinement to the URREF to capture handling of inconsistency.

One of the main threats to the underwater communication cables identified in the recent years is possible tampering or damage by malicious actors. This paper proposes a solution with explanation abilities to detect and investigate this kind of threat within the evidence theory framework. The reasoning scheme implements the traditional “opportunity-capability-intent” threat model to assess a degree to which a given vessel may pose a threat. The scenario discussed considers a variety of possible pieces of information available from different sources. A source quality model is used to reason with the partially reliable sources and the impact of this meta-information on the overall assessment is illustrated. Examples of uncertain relationships between the relevant variables are modelled and the constructed model is used to investigate the probability of threat of four vessels of different types. One of these cases is discussed in more detail to demonstrate the explanation abilities. Explanations about inference are provided thanks to sensitivity spaces in which the impact of the different pieces of information on the reasoning are compared.

To support situation awareness, the benefit of using a variety of sources is undeniable although it brings additional challenges related to possible conflicting information, heterogeneity in data formats, semantics, uncertainty types, and information quality. Information and source quality are intertwined concepts which assessments connect with the evaluation of uncertainty handling in information fusion solutions. While the Uncertainty Representation Reasoning Evaluation Framework (URREF) ontology focuses on assessment criteria, peripheral concepts still play a critical role. In this paper, we propose an Information and Source Quality (ISQ) ontology formalising the relationships between information-related concepts, and discuss information interpretation in support of Maritime Situation Awareness. Specifically, this paper links the concepts ofInformation Source, DatasetandPiece of Information, and connects them to the corresponding quality concepts. Such concepts link to the upper level concepts of the URREF ontologySource(of information) and data Quality. The ontology further expands to the uncertainty modelling and the algorithm design. We conclude on future work and identify future avenues, especially the extension to the formalisation of the evaluation process.

The choices of the uncertainty representations and reasoning methods have a critical impact on the development and deployment of modern fusion solutions. They influence the development effort, the quality of the resulting solutions as well as the deployment costs. However, such choices require an analysis that considers many operational and theoretical aspects. The Uncertainty Representation and Reasoning Evaluation Framework (URREF) concepts enable such an analysis. The proposed URREF -driven development approach establishes relations between the URREF criteria and evaluation subjects in the context of a development and deployment life cycle. In this way the relevant theoretical elements of the fusion techniques employed are emphasized and evaluated at various stages of the development process, facilitating informed design choices as well as systematic and tractable evaluation of complex fusion solutions. The concepts are illustrated through the assessment of a high-level fusion approach supporting estimation of the whereabouts of wildlife poachers.

S.192-197

The International Society of Information Fusion (ISIF) Evaluation Techniques for Uncertainty Representation Working Group (ETURWG) investigates the quantification and evaluation of all types of uncertainty regarding the inputs, reasoning and outputs of the information fusion process. The ETURWG is developing an Uncertainty Representation and Reasoning Framework (URREF) ontology for this purpose. This paper outlines a start towards the process of defining metrics for the URREF data criteria, which will align the URREF ontology with practical application. A criterion can be evaluated according to several metrics, and a metric can be applied to several criteria. As such, the ontology would have to reflect the nature of a many-to-many mapping between criteria and metrics. The main findings and suggestions of the paper advancing the use of URREF are: 1) The Weight of Information (WoI) is dependent on data criteria, which in turn depend on source criteria. 2) Criteria and metrics that apply to evidence (typically an input of the fusion system), could equally apply to the fusion system outputs or internal information, which in turn could form the inputs of another system. As such the word “Evidence” in the terms “Piece of Evidence” and “Weight of Evidence” should be replaced by the word “Information”. 3) Accuracy and precision and associated metrics are ubiquitous in the URREF ontology and can evaluate many parts of the fusion system. 4) The weight of information also assumes an important position in the ontology, as it depends on several source and data criteria.

The question addressed in this paper is “what” is to be evaluated by the Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology. We thus identify the elements composing uncertainty representation and reasoning approaches, which constitute various subjects being assessed. We distinguish between primary evaluation subjects (Uncertainty Representation and Reasoning components of the fusion algorithm), and secondary evaluation subjects (source of information, piece of information, fusion method and mathematical model). This paper proposes a list of source quality criteria to be added to the ontology and establishes formal links between the secondary and primary evaluation subjects. The key contribution of the paper is the update of the definitions of sub-criteria of the Expressiveness criterion together with suggestions for complementary concepts to be included in the ontology (type of scale, type of uncertainty expression). Conclusions are drawn to extend the work in using the expressiveness criterion for information fusion analysis.

Defence Research and Development Canada at Valcartier is pursuing the exploration of situation analysis concepts and the prototyping of computer-based decision support systems to maintain the state of situational awareness for the decision maker. The integration of the human element at the beginning of the analysis process is an important facet of our approach. The mathematical formalism and methodology proposed will be illustrated on concrete examples for visibility-based terrain analysis and reasoning for combat search and rescue (CSAR) operations. The work presented is based on the North Atlantis scenario GIS dataset, depicting a conflict taking place over an imaginary continent. The dataset is composed of topographic, hydrographical, transportation, and other typical land cover layers. Applications presented will include landing site determination as well as shortest path to crash site determination.

This paper details and deepens a previous work where the Interpreted Systems semantics was proposed as a general framework for situation analysis (SA). This framework is particularly efficient for representing and reasoning about knowledge and uncertainty when performing situation analysis tasks. Our approach of SA is to base our analysis on the production of state transition systems consisting in the set of all temporal trajectories possibly obtained upon the execution of a given set of agents' protocols. Thus seen, the SA task involves the definition of more or less subtle reasoning about graph structures. A formal situation analysis model is defined as an interpreted algorithmic belief change system. In such a model, the notions of situation, situation awareness and situation analysis are provided. The analysis of the situation is done through the verification of implicit notions of knowledge with temporal properties. Implicit knowledge is distinguished from explicit knowledge and situation awareness is defined in terms of the computing power of resource-bounded agents. A general plausibility measure allows us to model belief while making the link with quantitative representations of uncertainty such as probabilities, belief functions and possibilities. The propsed modelisation of the Situation Analysis process, while compatible with the traditionnal implicit representation of knowledge found in modal logic, allows us to link the decision processes of the agents, their awareness of the situation with the observations they make about the environment.

This paper proposes a discussion on the formal links between the situation calculus and the semantics of interpreted systems as far as they relate to higher-level information fusion tasks. Among these tasks situation analysis require to be able to reason about the decision processes of coalitions. Indeed in higher levels of information fusion, one not only need to know that a certain proposition is true (or that it has a certain numerical measure attached), but rather needs to model the circumstances under which this validity holds as well as agents' properties and constraints. In a previous paper the authors have proposed to use the interpreted system semantics as a potential candidate for the unification of all levels of information fusion. In the present work we show how the proposed framework allow to bind reasoning about courses of action and situation awareness. We propose in this paper a (1) model of coalition, (2) a model of ability in the situation calculus language and (3) a model of situation awareness in the interpreted systems semantics. Combining the advantages of both situation calculus and the interpreted systems semantics, we show how the situation calculus can be framed into the interpreted systems semantics. We illustrate on the example of RAP compilation in a coalition context, how ability and situation awareness interact and what benefit is gained. Finally, we conclude this study with a discussion on possible future works.