Test and Evaluation for verification and validation (V&V) of sensor data fusion techniques utilize methods of uncertainty analysis. The Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology identifies many attributes of metrics (i.e., semantic meaning, object metrics, and subjective quality). With the growing interest in artificial intelligence (AI) due to large data corpus access, fast compute power, and machine/deep learning (ML/DL) techniques; V&V of these methods are needed. In this paper, the enhancement of the URREF to utilize a risk assessment for decision is demonstrated towards analysis/alignment of ML/DL methods that utilize multi-modal data fusion. Evidential reasoning is considered in the use case to provide data handing reliability source and processing credibility to measure decision risk in a maritime domain awareness scenario.

ChatGPT, OpenAI’s chatbot, has gained consider-able attention since its launch in November 2022, owing to its ability to formulate articulated responses to text queries and comments relating to seemingly any conceivable subject. As impressive as the majority of interactions with ChatGPT are, this large language model has a number of acknowledged shortcomings, which in several cases, may be directly related to how ChatGPT handles uncertainty. The objective of this paper is to pave the way to formal analysis of ChatGPT uncertainty handling. To this end, the ability of the Uncertainty Representation and Reasoning Framework (URREF) ontology is assessed, to support such analysis. Elements of structured experiments for reproducible results are identified. The dataset built varies Information Criteria of Correctness, Non-specificity, Self-confidence, Relevance and Inconsistency, and the Source Criteria of Reliability, Competency and Type. ChatGPT’s answers are analyzed along Information Criteria of Correctness, Non-specificity and Self-confidence. Both generic and singular information are sequentially provided. The outcome of this preliminary study is twofold: Firstly, we validate that the experimental setup is efficient in capturing aspects of ChatGPT uncertainty handling. Secondly, we identify possible modifications to the URREF ontology that will be discussed and eventually implemented in URREF ontology Version 4.0 under development.

AI-based decision support solutions require life cycles that adequately address critical steps, such as (i) finding suitable machine learning (ML) methods for the problem at hand, (ii) preparing and executing adequate data acquisition processes and (iii) tractable evaluation of the overall solution. Understanding the data generating processes is key in achieving this. Training and test data can be seen as a result of a causal data generation process, a sampling process in which the data is collected from different sources that are influenced by multiple interdependent phenomena. This is represented by a Qualitative Model of Data Generation Processes (QM-DGP), a causal graphical model. QM-DGP facilitates analysis of the complexity of the underlying data generating processes that can inform the development of trustable ML-based solutions in multiple ways. Firstly, this analysis is the basis for the determination of the required complexity of the ML models. Secondly, it facilitates the determination of the quantities of training data supporting good learning results. Thirdly, it can provide guidance for a systematic simplification of the models, supporting tractable solutions without significantly reduced performance. The construction of QM-DGP and the analysis benefit from sound theoretical concepts, such as d-separation and I-Maps. Experimental results with simulated data indicate that the approach can be effective in predicting the required quantities of training data and the determination of the modelling complexity using different types of models.

Video data is increasingly being provided by multiple sources. These sources are used to obtain reliable feature information within the context of training data shortages. As such it becomes crucial to interpret and refine such information. Recent research on video content analysis often use deep-learning features due to their outstanding performance in different domains. These features are aggregated over time to create a video-level descriptor. In this research, we explore the potential of statistical feature aggregation methods in combining deep-learning features that represent the visual content of videos. In particular, the contributions of this paper are two-fold: Firstly, we compared statistical feature aggregation methods by selecting movie sequels, calculating their in-sequel-mean-distance and out-of-sequel-mean-distance as well as the Bhattacharyya distance between them and using Principal Component Analysis (PCA) and T-distributed Stochastic Neighbour Embedding (t-SNE) to visualise their distribution in the feature space. This is important to easily comprehend the video content without the need to watch them and understand how well the statistical feature aggregation methods represent the videos. Secondly, the performance of these aggregation methods is explored in the context of a content-based video retrieval task which is evaluated in terms of relevance. The observed results show that the statistical feature aggregation method based on variance outperforms the methods based on maximum, mean, median, median absolute deviation and interquartile range. This outcome is supported when interpreting the t-SNE visualisation method as well as Bhattacharyya distance calculations. Overall statistical feature aggregation methods which measure the spread of a distribution achieve better performance compared to methods that are a measure of location.

Tracking of multiple extended three-dimensional targets in the presence of partial occlusions from point cloud measurement sets remains a challenge but has obvious applications in autonomous vehicles. We demonstrate that existing range-image based approaches to point cloud segmentation and motion detection can be fused with fast point cloud registration techniques to aid track association. Consequently, multiple generic extended targets in point clouds can be tracked. We also show that robust occlusion handling can be achieved using temporally accumulated geometric information from co-registered point clouds in a static local reference frame. The occlusion logic is based on a novel probabilistic occlusion reasoning approach combined with a simulated geometric handling of self-occlusions. The proposed tracking pipeline was tested on synthetic target geometries superimposed on real-world backgrounds, as well as against measured data from benchmark autonomous driving datasets. The targets tracked encountered both artificially introduced and naturally occurring foreground occlusions as well as self-occlusions. Results show notable robustness to foreground occlusions. Incorrect track associations are possible when multiple targets in proximity display similar dynamics and possess nearly identical geometries. Our sequence of tracking algorithms relies on multiple tunable hyper-parameters. These parameters require further automated runtime-optimization before robust, real-time field application can be achieved.

In this paper, data from a simulated data set were fused with a significantly smaller measured data set for the purpose of training a deep neural network to identify animals from the footage of a dash cam. The trained networks were used to detect wildlife in an environment similar to game reserves in South Africa. To enable the automatic collection of data for the experiment, a simulated environment was created to simulate four classes of wildlife found in South Africa: buffalo, elephants, rhino and zebra. The network structure for the detector network selected was an adapted version of the tiny YOLOv3 network. It was discovered that using transfer learning and fine tuning resulted in two models with higher accuracy of 82.59% and 86.64% [email protected] respectively than models where no transfer learning was used. The results were achieved when tested on a testing set of digital photographic images. These networks, initialised using transfer learning, were also faster and easier to train than training using a combined data set of photographic and simulated images from scratch. The simulated environment can however not replace real-life data, as was proven by an accuracy of no better than chance for the model trained using only simulated data.

In this paper we explore automatic genre labelling of motion picture previews using audio-visual features present in movie trailers and the focus is on fusion techniques (early fusion and late fusion) and the resultant improvement on classification accuracy. This paper proposes a novel combination of deep learned features (from a pretrained VGG-16 model) obtained using a state-of-the-art shot detector and hand-crafted audio features. This combination of features and an associated comparison of early and late fusion with these features has not been attempted in the literature before. Furthermore, two popular fusion techniques and three distinct classification algorithms are investigated to determine the optimal fusion technique and classifier combination. The study uses a subset of the LMTD-9 movie trailer dataset with selected genres (action, comedy, drama and horror). The best performing low-level audio features are comprised of timbre features extracted using the MIRtoolbox followed by standalone mel-frequency cepstral coefficients. The best performing high-level audio feature is tonality. Audio features are augmented by visual features extracted using a pre-trained convolutional neural network (VGG-16). Feature fusion (early and late fusion) methods are investigated together with classification methods such as extreme gradient boosting, support vector machine and a neural network. Evaluation metrics such as precision, recall, confusion matrices and F1 score are used to measure classification accuracy. Early fusion methods outperform late fusion methods with a classification performance gain of approximately 10% for a four class classification problem. The best classification performance for early fusion obtained with a support vector machine is (73.12% accuracy), followed by the extreme gradient boosting classifier (69.37% accuracy) and neural network classifier (67.50% accuracy), whereas chance is 25%. It is shown that superior classification performance can be achieved by employing early feature fusion of low-level audio descriptors, high-level audio descriptors and high-level visual feature descriptors together with suitable classifiers.

Systems for threat analysis enable users to understand the nature and behavior of threats and to undertake a deeper analysis for detailed exploration of threat profile and risk estimation. Models for threat analysis require significant resources to be developed and are often relevant to limited application tasks. This paper investigated the implicit and explicit uncertainty assessments to be taken into account for threat analysis systems to be effective for providing a relevant threat characterization. The intent of this paper is twofold. The first is to present and discuss an approach to define a model for cyber threats within a simplified expert model and to translate it into a Bayesian network as a tool for the development of practical scenarios for cyber threats analysis. The second is to address the question of assessing the Bayesian network build and its intrinsic knowledge representation model and to show how modeling decisions impact the outcome of the system. The paper describes the construction of an expert model and the corresponding BN to analyze cyber threats, investigates various types of induced uncertainty with the URREF criteria simplicity and expressiveness and implements an assessment procedure to evaluate the overall approach.

Sensormanagement is an important function of any data fusion center as the output of a fusion system is dependent on the quality of the information collected. In this paper, the scheduling aspect of sensor management function is implemented using Response Surface Modeling (RSM). Applying RSM requires formulating the sensor management function as an objective function. The benefit of RSM over prior global optimization approaches is the simplification of the evaluation of this objective function to find global optima. This leads to either reduced computational requirements and/ or shorter due times for creating sensor schedules. This work shows the utility of RSM towards scheduling multiple sensors, and seeks to introduce RSM to the sensor management community. It is shown that the RSM scheduler provides a significant improvement towards reducing the number of missed targets in a surveillance radar network. This is compared to performing a uniform scanning regime (or sequential stepped scan) often employed. Very few iterations are required to provide this gain. The RSM technique also quickly determines where the most effective use of sensor resources needs to be applied. Consequently, it spends more radar dwell time on these beam locations.

This paper presents a Bayesian tracking approach that exploits various types of context information. The filtering accuracy and precision are improved by using uncertain information about (i) the constraints on the target mobility, (ii) environmental influences on the sensor performance and (iii) typical target behaviors. The approach combines particle filters with exact Bayesian networks. The overall process is equivalent to approximate inference on elaborate dynamic Bayesian networks that systematically capture non-trivial correlations between the estimated states of the dynamic processes, the associated observations and the various factors influencing the dynamic processes. The particle filter supports reasoning about continuous dynamic processes spanning large areas, while the Bayesian networks are used for the implementation of advanced sensor models and for the fusion of uncertain data on mobility constraints. The derivation of the solution is based on the decomposability principles of Bayesian networks. The approach is illustrated with the help of a challenging wildlife protection application. A set of qualitative experiments shows the improvement in tracking performance by considering the different types of context information.

The choices of the uncertainty representations and reasoning methods have a critical impact on the development and deployment of modern fusion solutions. They influence the development effort, the quality of the resulting solutions as well as the deployment costs. However, such choices require an analysis that considers many operational and theoretical aspects. The Uncertainty Representation and Reasoning Evaluation Framework (URREF) concepts enable such an analysis. The proposed URREF -driven development approach establishes relations between the URREF criteria and evaluation subjects in the context of a development and deployment life cycle. In this way the relevant theoretical elements of the fusion techniques employed are emphasized and evaluated at various stages of the development process, facilitating informed design choices as well as systematic and tractable evaluation of complex fusion solutions. The concepts are illustrated through the assessment of a high-level fusion approach supporting estimation of the whereabouts of wildlife poachers.

The International Society of Information Fusion (ISIF) Evaluation Techniques for Uncertainty Representation Working Group (ETURWG) investigates the quantification and evaluation of all types of uncertainty regarding the inputs, reasoning and outputs of the information fusion process. The ETURWG is developing an Uncertainty Representation and Reasoning Framework (URREF) ontology for this purpose. This paper outlines a start towards the process of defining metrics for the URREF data criteria, which will align the URREF ontology with practical application. A criterion can be evaluated according to several metrics, and a metric can be applied to several criteria. As such, the ontology would have to reflect the nature of a many-to-many mapping between criteria and metrics. The main findings and suggestions of the paper advancing the use of URREF are: 1) The Weight of Information (WoI) is dependent on data criteria, which in turn depend on source criteria. 2) Criteria and metrics that apply to evidence (typically an input of the fusion system), could equally apply to the fusion system outputs or internal information, which in turn could form the inputs of another system. As such the word “Evidence” in the terms “Piece of Evidence” and “Weight of Evidence” should be replaced by the word “Information”. 3) Accuracy and precision and associated metrics are ubiquitous in the URREF ontology and can evaluate many parts of the fusion system. 4) The weight of information also assumes an important position in the ontology, as it depends on several source and data criteria.

The question addressed in this paper is “what” is to be evaluated by the Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology. We thus identify the elements composing uncertainty representation and reasoning approaches, which constitute various subjects being assessed. We distinguish between primary evaluation subjects (Uncertainty Representation and Reasoning components of the fusion algorithm), and secondary evaluation subjects (source of information, piece of information, fusion method and mathematical model). This paper proposes a list of source quality criteria to be added to the ontology and establishes formal links between the secondary and primary evaluation subjects. The key contribution of the paper is the update of the definitions of sub-criteria of the Expressiveness criterion together with suggestions for complementary concepts to be included in the ontology (type of scale, type of uncertainty expression). Conclusions are drawn to extend the work in using the expressiveness criterion for information fusion analysis.