In human-machine teaming, machine capability to automatically check human teammates for their Situation Awareness (SA) will not only improve human-machine teaming performance, but it will also create opportunities to provide timely interventions. Such machine capability may benefit a wide range of applications, ranging from Advanced Driver Assistance Systems (ADAS) to highly complex warfare situations. This paper extends our previous work on operator SA assessment (FUSION 2022 paper) by leveraging state-of-the-art Computer Vision (CV) techniques on object detection, segmentation and multi-target tracking (or multi-object tracking) for identifying relevant Situation Elements (SEs) and extracting information necessary for calculating Probability of Attending for those SEs. First, YOLOv7 and StrongSort is used for automated detection and tracking of SEs identified from video streams. Then, instance segmentation is utilized for identifying relevant pixel information of SEs (i.e., location, size, color) for computing specific Visual Attention Allocation (VAA) values of probability of attending (P(SE)) and attention allocation proportion ($f_{i}$) via the use of the SEEV (i.e., Salience, Effort, Expectancy, and Value) SA model. Finally, operator SA is derived by analyzing the operator’s visual attention via the use of eye-tracking to check if the operator is paying attention to relevant SEs. This paper concludes with a discussion of our current progress, technical challenges being pursued and potential future extensions.

Epistemic decisions about which sources to trust and query are critical for a decision-maker, when the end-goal decisions are to be made using limited resources. Toward this, we previously proposed preliminary extensions to classical measures of Value of Information (VoI) for imprecise belief states represented by belief functions relying on a general observation model. These methods primarily aim to assist a decision-maker toward making rational decisions, by generating necessary metadata about the information that is being considered for the decision-making task (e.g. probability of source reliability, degree of self-confidence expressed by the source). In this paper, we explore the behavior and performance of our previously proposed belief theoretic VoI measures, the Evidential Expecetd Value of Sampled Information (EEVSI), and propose a procedure to sequentially select sources to query. We also consider the case where the information sources are providing contradictory evidence. We leverage a maritime surveillance scenario, where the decision-maker has to make a rational ordered selection of information sources among a set of both physical sensors and human sources, to illustrate the behavior of the proposed method. We compare the proposed policy with a traditional probability-based approach in a simulation environment. We conclude by providing some insights on future research directions to further expand on our proposed new measures.

This paper describes a multiple-hypothesis tracking (MHT) formulation of a particular set of situational awareness problems that involve monitoring of spatio-temporal phenomena using ubiquitous sensing. In particular, the focus is on large-scale monitoring (or tracking) applications utilizing volunteer mobile ubiquitous sensors to track the evolution of spatio-temporal `targets' of interest (e.g., tracking snow-fall or ride-quality at a certain stretch of a roadway using vehicles). An efficient framework is developed utilizing MHT as the basis to carry out detection and tracking of evolutionary behavior. The framework is described via an illustrative example on ride-quality monitoring as applied to autonomous driving environments, where vibration and GPS data recorded by voluntarily participating vehicles are utilized for threat detection and tracking.

Multi-sensor radar tracking systems can provide better system performance, visibility and complementary information than single sensor tracking systems. One way to utilize multiple sensors for target tracking is to generate individual sensor detections and fuse these detections to produce optimized system tracks. All or a subset of detections from the sensors are fed into a tracker algorithm, allowing the filter to choose the best possible measurement association. This paper discusses the implementation of a multi-sensor target tracking system using three different sensor configurations for providing vehicle detections. The three sensor configurations compare the accuracy of one vs two radar sensors, operating in either a stepped frequency waveform or a hybrid stepped frequency and continuous waveform. The purpose of this paper is to provide insight into the possibilities of multi-sensor radar tracking for the advancement of autonomous driving.

Robust belief revision methods are crucial in streaming data situations for updating existing knowledge (or beliefs) with new incoming evidence. Bayes conditioning is the primary mechanism in use for belief revision in data fusion systems that use probabilistic inference. However, traditional conditioning methods face several challenges due to inherent data/source imperfections in big-data environments that harness soft (i.e., human or human-based) sources in addition to hard (i.e., physics-based) sensors. The objective of this paper is to investigate the most natural extension of Bayes conditioning that is suitable for evidence updating in the presence of such uncertainties. By viewing the evidence updating process as a thought experiment, an elegant strategy is derived for robust evidence updating in the presence of extreme uncertainties that are characteristic of big-data environments. In particular, utilizing the Fagin-Halpern conditional notions, a natural extension to Bayes conditioning is derived for evidence that takes the form of a general belief function. The presented work differs fundamentally from the Conditional Update Equation (CUE) and authors own extensions of it. An overview of this development is provided via illustrative examples. Furthermore, insights into parameter selection under various fusion contexts are also provided.

The unintentional transport of invasive species (i.e., non-native and harmful species that adversely affect habitats and native species) through the Global Shipping Network (GSN) causes substantial losses to social and economic welfare (e.g., annual losses due to ship-borne invasions in the Laurentian Great Lakes is estimated to be as high as USD 800 million). Despite the huge negative impacts, management of such invasions remains challenging because of the complex processes that lead to species transport and establishment. Numerous difficulties associated with quantitative risk assessments (e.g., inadequate characterizations of invasion processes, lack of crucial data, large uncertainties associated with available data, etc.) have hampered the usefulness of such estimates in the task of supporting the authorities who are battling to manage invasions with limited resources. We present here an approach for addressing the problem at hand via creative use of computational techniques and multiple data sources, thus illustrating how data mining can be used for solving crucial, yet very complex problems towards social good. By modeling implicit species exchanges as a network that we refer to as the Species Flow Network (SFN), large-scale species flow dynamics are studied via a graph clustering approach that decomposes the SFN into clusters of ports and inter-cluster connections. We then exploit this decomposition to discover crucial knowledge on how patterns in GSN affect aquatic invasions, and then illustrate how such knowledge can be used to devise effective and economical invasive species management strategies. By experimenting on actual GSN traffic data for years 1997-2006, we have discovered crucial knowledge that can significantly aid the management authorities.

Automated Collaborative Filtering (ACF) refers to a group of algorithms used in recommender systems, a research topic that has received considerable attention due to its e-commerce applications. However, existing techniques are rarely capable of dealing with imperfections in user-supplied ratings. When such imperfections (e.g., ambiguities) cannot be avoided, designers resort to simplifying assumptions that impair the system's performance and utility. We have developed a novel technique referred to as CoFiDS—Collaborative Filtering based on Dempster-Shafer belief-theoretic framework—that can represent a wide variety of data imperfections, propagate them throughout the decision-making process without the need to make simplifying assumptions, and exploit contextual information. With its DS-theoretic predictions, the domain expert can either obtain a "hard” decision or can narrow the set of possible predictions to a smaller set. With its capability to handle data imperfections, CoFiDS widens the applicability of ACF to such critical and sensitive domains as medical decision support systems and defense-related applications. We describe the theoretical foundation of the system and report experiments with a benchmark movie data set. We explore some essential aspects of CoFiDS' behavior and show that its performance compares favorably with other ACF systems.