In human-machine teaming, machine capability to automatically check human teammates for their Situation Awareness (SA) will not only improve human-machine teaming performance, but it will also create opportunities to provide timely interventions. Such machine capability may benefit a wide range of applications, ranging from Advanced Driver Assistance Systems (ADAS) to highly complex warfare situations. This paper extends our previous work on operator SA assessment (FUSION 2022 paper) by leveraging state-of-the-art Computer Vision (CV) techniques on object detection, segmentation and multi-target tracking (or multi-object tracking) for identifying relevant Situation Elements (SEs) and extracting information necessary for calculating Probability of Attending for those SEs. First, YOLOv7 and StrongSort is used for automated detection and tracking of SEs identified from video streams. Then, instance segmentation is utilized for identifying relevant pixel information of SEs (i.e., location, size, color) for computing specific Visual Attention Allocation (VAA) values of probability of attending (P(SE)) and attention allocation proportion ($f_{i}$) via the use of the SEEV (i.e., Salience, Effort, Expectancy, and Value) SA model. Finally, operator SA is derived by analyzing the operator’s visual attention via the use of eye-tracking to check if the operator is paying attention to relevant SEs. This paper concludes with a discussion of our current progress, technical challenges being pursued and potential future extensions.

Epistemic decisions about which sources to trust and query are critical for a decision-maker, when the end-goal decisions are to be made using limited resources. Toward this, we previously proposed preliminary extensions to classical measures of Value of Information (VoI) for imprecise belief states represented by belief functions relying on a general observation model. These methods primarily aim to assist a decision-maker toward making rational decisions, by generating necessary metadata about the information that is being considered for the decision-making task (e.g. probability of source reliability, degree of self-confidence expressed by the source). In this paper, we explore the behavior and performance of our previously proposed belief theoretic VoI measures, the Evidential Expecetd Value of Sampled Information (EEVSI), and propose a procedure to sequentially select sources to query. We also consider the case where the information sources are providing contradictory evidence. We leverage a maritime surveillance scenario, where the decision-maker has to make a rational ordered selection of information sources among a set of both physical sensors and human sources, to illustrate the behavior of the proposed method. We compare the proposed policy with a traditional probability-based approach in a simulation environment. We conclude by providing some insights on future research directions to further expand on our proposed new measures.

This paper describes a multiple-hypothesis tracking (MHT) formulation of a particular set of situational awareness problems that involve monitoring of spatio-temporal phenomena using ubiquitous sensing. In particular, the focus is on large-scale monitoring (or tracking) applications utilizing volunteer mobile ubiquitous sensors to track the evolution of spatio-temporal `targets' of interest (e.g., tracking snow-fall or ride-quality at a certain stretch of a roadway using vehicles). An efficient framework is developed utilizing MHT as the basis to carry out detection and tracking of evolutionary behavior. The framework is described via an illustrative example on ride-quality monitoring as applied to autonomous driving environments, where vibration and GPS data recorded by voluntarily participating vehicles are utilized for threat detection and tracking.

Multi-sensor radar tracking systems can provide better system performance, visibility and complementary information than single sensor tracking systems. One way to utilize multiple sensors for target tracking is to generate individual sensor detections and fuse these detections to produce optimized system tracks. All or a subset of detections from the sensors are fed into a tracker algorithm, allowing the filter to choose the best possible measurement association. This paper discusses the implementation of a multi-sensor target tracking system using three different sensor configurations for providing vehicle detections. The three sensor configurations compare the accuracy of one vs two radar sensors, operating in either a stepped frequency waveform or a hybrid stepped frequency and continuous waveform. The purpose of this paper is to provide insight into the possibilities of multi-sensor radar tracking for the advancement of autonomous driving.

Automotive LiDAR and Radar sensor fusion is quickly developing as a technique used to provide ground truth data for radar development. Radar sensors in an automotive application are known to have many sources of noise such as bumper fascia distortion, non-linear component behavior plus angle and range resolution error from known Additive Gaussian White Noise sources in the Radio Frequency front end of the radar. State of the art LiDAR sensors such as the Velodyne HDL-32E and HDL-64E can be mounted outside of the bumper fascia in a test vehicle configuration and are not subject to the same amount of internal noise as an RF system. This means that LiDAR point cloud data can be used for reference system measurements in a possible data fusion system that can be used to characterize radar accuracy and performance. Representative LiDAR and Radar track data was collected and then was input into multiple instances of a Kalman filter that provided estimates used as a basis for comparison between the two modalities.

Lack of comprehensive and up-to-date information on degrading road conditions is only exacerbating the already inefficient, expensive and labor-expensive methods associated with road maintenance and repair. Among many sensing modalities that are currently being explored, MEMS accelerometers are increasingly becoming popular in the task of road quality assessment, especially in large-scale crowdsourced data gathering efforts. In this paper, recent results on the use of accelerometers for detection of road abnormalities is presented. In particular, with an emphasis on developing a low-cost yet robust road abnormality detection system, predictive effectiveness offeaturesgenerated from acceleration sensors is explored. By utilizing a signal transmission model that captures the system dynamics and low-pass filtering effects associated with vehicle suspension system, a simplified approach towards the `reconstruction' of road surface conditions from vertical acceleration measurements is presented. Furthermore, with this particular signal transmission model in place, predictive effectiveness of features derived from acceleration sensors is explored via the use of a statistical approach referred to asrelief algorithm. The signal model and feature analysis approach is demonstrated via areal-lifedataset.

Robust belief revision methods are crucial in streaming data situations for updating existing knowledge (or beliefs) with new incoming evidence. Bayes conditioning is the primary mechanism in use for belief revision in data fusion systems that use probabilistic inference. However, traditional conditioning methods face several challenges due to inherent data/source imperfections in big-data environments that harness soft (i.e., human or human-based) sources in addition to hard (i.e., physics-based) sensors. The objective of this paper is to investigate the most natural extension of Bayes conditioning that is suitable for evidence updating in the presence of such uncertainties. By viewing the evidence updating process as a thought experiment, an elegant strategy is derived for robust evidence updating in the presence of extreme uncertainties that are characteristic of big-data environments. In particular, utilizing the Fagin-Halpern conditional notions, a natural extension to Bayes conditioning is derived for evidence that takes the form of a general belief function. The presented work differs fundamentally from the Conditional Update Equation (CUE) and authors own extensions of it. An overview of this development is provided via illustrative examples. Furthermore, insights into parameter selection under various fusion contexts are also provided.

The unintentional transport of invasive species (i.e., non-native and harmful species that adversely affect habitats and native species) through the Global Shipping Network (GSN) causes substantial losses to social and economic welfare (e.g., annual losses due to ship-borne invasions in the Laurentian Great Lakes is estimated to be as high as USD 800 million). Despite the huge negative impacts, management of such invasions remains challenging because of the complex processes that lead to species transport and establishment. Numerous difficulties associated with quantitative risk assessments (e.g., inadequate characterizations of invasion processes, lack of crucial data, large uncertainties associated with available data, etc.) have hampered the usefulness of such estimates in the task of supporting the authorities who are battling to manage invasions with limited resources. We present here an approach for addressing the problem at hand via creative use of computational techniques and multiple data sources, thus illustrating how data mining can be used for solving crucial, yet very complex problems towards social good. By modeling implicit species exchanges as a network that we refer to as the Species Flow Network (SFN), large-scale species flow dynamics are studied via a graph clustering approach that decomposes the SFN into clusters of ports and inter-cluster connections. We then exploit this decomposition to discover crucial knowledge on how patterns in GSN affect aquatic invasions, and then illustrate how such knowledge can be used to devise effective and economical invasive species management strategies. By experimenting on actual GSN traffic data for years 1997-2006, we have discovered crucial knowledge that can significantly aid the management authorities.

In a multi-agent data fusion scenario, agents may iteratively exchange their states to arrive at a consensus state which signifies ‘general agreement’ among the agents. Agent states that are being exchanged may have been generated from hard (i.e., physics based) or soft (i.e., human based evidence. such as opinions or beliefs regarding an event) sensors. Convergence analysis becomes an extremely challenging problem in such complex fusion environments, which may involve communication delays, ad-hoc paths, etc. In this paper, we analyze consensus of a Dempster-Shafer theoretic (DST) fusion operator by formulating the consensus problem as finding common fixed points of a pool of paracontracting operators. Due to its DST basis, this consensus protocol can deal with a wider variety of data imperfections characteristic of hard+soft data fusion environments. It also easily adapts itself to networks where agent states are captured with probability mass functions because they can be considered a special case of DST models.

Dempster-Shafer (DS) belief theory provides a convenient framework for the development of powerful data fusion engines by allowing for a convenient representation of a wide variety of data imperfections. The recent work on the DS theoretic (DST) conditional approach, which is based on the Fagin-Halpern (FH) DST conditionals, appears to demonstrate the suitability of DS theory for incorporating both soft (generated by human-based sensors) and hard (generated by physics-based sources) evidence into the fusion process. However, the computation of the FH conditionals imposes a significant computational burden. One reason for this is the difficulty in identifying the FH conditional core, i.e., the set of propositions receiving nonzero support after conditioning. The conditional core theorem (CCT) in this paper redresses this shortcoming by explicitly identifying the conditional focal elements with no recourse to numerical computations, thereby providing a complete characterization of the conditional core. In addition, we derive explicit results to identify those conditioning propositions that may have generated a given conditional core. This "converse" to the CCT is of significant practical value for studying the sensitivity of the updated knowledge base with respect to the evidence received. Based on the CCT, we also develop an algorithm to efficiently compute the conditional masses (generated by FH conditionals), provide bounds on its computational complexity, and employ extensive simulations to analyze its behavior.

In many contexts, one is confronted with the problem of extract ing information from large amounts of different types soft data (e.g., text) and hard data (from e.g., physics-based sensing systems). In handling hard data, signal and data processing offers a wealth of methods related to modeling, estimation, tracking, and inference tasks. However, soft data present several challenges that necessitate the development of new data processing methods. For example, with suitable statistical natural language processing (NLP) methods, text can be converted into logic statements that are associated with various forms of associated uncertainty related to the credibility of the statement, the reliability of the text source, and so forth. In combining or fusing soft data with either soft or hard data, one must deploy methods that can suitably preserve and update the uncertainty associated with the data, thereby providing uncertainty bounds related to any inferences regarding semantics. Since standard Bayesian probabilistic approaches have problems with suitably handling uncertain logic statements, there is an emerging need for new methods for processing heterogeneous data. In this paper, we describe a framework for fusing soft and hard data based on the Dempster-Shafer (DS) belief theoretic approach which is well-suited to the task of capturing the types of models and uncertain rules that are more typical of soft data. Since the effectiveness of traditional DS methods has been hampered by high computational requirements, we base the processing framework on our new conditional approach to DS theoretic evidence updating and fusion. We address the issue of laying the foundation for a theoretically justifiable, and computationally efficient framework for fusing soft and hard data taking into account the inherent data uncertainty such as reliability and credibility. Moreover, we present an illustrative ex ample that highlights the potential for the DS conditional approach for fusing heterogeneous data.

Automated Collaborative Filtering (ACF) refers to a group of algorithms used in recommender systems, a research topic that has received considerable attention due to its e-commerce applications. However, existing techniques are rarely capable of dealing with imperfections in user-supplied ratings. When such imperfections (e.g., ambiguities) cannot be avoided, designers resort to simplifying assumptions that impair the system's performance and utility. We have developed a novel technique referred to as CoFiDS—Collaborative Filtering based on Dempster-Shafer belief-theoretic framework—that can represent a wide variety of data imperfections, propagate them throughout the decision-making process without the need to make simplifying assumptions, and exploit contextual information. With its DS-theoretic predictions, the domain expert can either obtain a "hard” decision or can narrow the set of possible predictions to a smaller set. With its capability to handle data imperfections, CoFiDS widens the applicability of ACF to such critical and sensitive domains as medical decision support systems and defense-related applications. We describe the theoretical foundation of the system and report experiments with a benchmark movie data set. We explore some essential aspects of CoFiDS' behavior and show that its performance compares favorably with other ACF systems.