• An IMM filter combines a static camera motion model with a dynamic model. • Ground-plane- and image-plane-based association scores are dynamically mixed. • Dynamic noise estimation is employed. • IMM-JHSE uses a single initial homography estimate, and 2D bounding boxes. • IMM-JHSE compares favourably to other methods, including 3D MOT methods. A novel online MOT algorithm, IMM Joint Homography State Estimation (IMM-JHSE), is proposed. IMM-JHSE uses an initial homography estimate as the only additional 3D information, whereas other 3D MOT methods use regular 3D measurements. By jointly modelling the homography matrix and its dynamics as part of track state vectors, IMM-JHSE removes the explicit influence of camera motion compensation techniques on predicted track position states, which was prevalent in previous approaches. Expanding upon this, static and dynamic camera motion models are combined using an IMM filter. A simple bounding box motion model is used to predict bounding box positions to incorporate image plane information. In addition to applying an IMM to camera motion, a non-standard IMM approach is applied where bounding-box-based BIoU scores are mixed with ground-plane-based Mahalanobis distances in an IMM-like fashion to perform association only, making IMM-JHSE robust to motion away from the ground plane. Finally, IMM-JHSE makes use of dynamic process and measurement noise estimation techniques. IMM-JHSE improves upon related techniques, including UCMCTrack, OC-SORT, C-BIoU and ByteTrack on the DanceTrack and KITTI-car datasets, increasing HOTA by 2.64 and 2.11, respectively, while offering competitive performance on the MOT17, MOT20 and KITTI-pedestrian datasets. Using publicly available detections, IMM-JHSE outperforms almost all other 2D MOT methods and is outperformed only by 3D MOT methods—some of which are offline—on the KITTI-car dataset. Compared to tracking-by-attention methods, IMM-JHSE shows remarkably similar performance on the DanceTrack dataset and outperforms them on the MOT17 dataset. The code is publicly available: https://github.com/Paulkie99/imm-jhse .

Visual odometry is an essential component of vision-based robotic navigation systems. A primary limitation of existing visual odometry solutions is their inability to achieve satisfactory performance in both high- and low-textured regions. In this paper, a robust RGB-D visual odometry method is proposed that fuses point and edge features. By combining the descriptiveness of feature points with the structure provided by edge data, a method that is robust to low-textured scenes is developed. Edge features are first detected and grouped based on the Gestalt principles of continuity and proximity. Edge groups are then associated between the current and previous frames using point features in the vicinity of the edges. Pose estimation is thereafter performed by first matching points between associated edge groups, filtering these points based on structural constraints imposed by the edges, and estimating the motion of the agent. Compared to state-of-the-art alternatives, such as REVO, MSC-VO, DROID-VO and SplaTAM on the TUM RGB-D, ICL-NUIM and Tartan-Air datasets, the resulting method reduces the root mean square absolute trajectory error, and translational and rotational relative pose errors by up to 58%, 75%, and 82%, respectively. This indicates that our method is not only more accurate than current approaches, but also more consistent, especially in low-structured and low-textured environments. • A method by which edge pixels can be grouped and matched between frames. • Localization by fusing the structure of edges with point feature descriptiveness. • Robust pose estimation in low-textured and low-structured environments. • Improved pose estimation accuracy compared to state-of-the-art alternatives.

A novel Bayesian framework is proposed, which explicitly relates the homography of one video frame to the next through an affine transformation while explicitly modelling keypoint uncertainty. The literature has previously used differential homography between subsequent frames, but not in a Bayesian setting. In cases where Bayesian methods have been applied, camera motion is not adequately modelled, and keypoints are treated as deterministic. The proposed method, Bayesian Homography Inference from Tracked Keypoints (BHITK), employs a two-stage Kalman filter and significantly improves existing methods. Existing keypoint detection methods may be easily augmented with BHITK. It enables less sophisticated and less computationally expensive methods to outperform the state-of-the-art approaches in most homography evaluation metrics. Furthermore, the homography annotations of the WorldCup and TS-WorldCup datasets have been refined using a custom homography annotation tool that has been released for public use. The refined datasets are consolidated and released as the consolidated and refined WorldCup (CARWC) dataset.

This study explores the feasibility of constructing a small-scale speech recognition system capable of competing with larger, modern automated speech recognition (ASR) systems in both performance and word error rate (WER). Our central hypothesis posits that a compact transformer-based ASR model can yield comparable results, specifically in terms of WER, to traditional ASR models while challenging contemporary ASR systems that boast significantly larger computational sizes. The aim is to extend ASR capabilities to under-resourced languages with limited corpora, catering to scenarios where practitioners face constraints in both data availability and computational resources. The model, comprising a compact convolutional neural network (CNN) and transformer architecture with 2.214 million parameters, challenges the conventional wisdom that large-scale transformer-based ASR systems are essential for achieving high accuracy. In comparison, contemporary ASR systems often deploy over 300 million parameters. Trained on a modest dataset of approximately 3000 h—significantly less than the 50,000 h used in larger systems—the proposed model leverages the Common Voice and LibriSpeech datasets. Evaluation on the LibriSpeech test-clean and test-other datasets produced character error rates (CERs) of 6.40% and 16.73% and WERs of 16.03% and 35.51% respectively. Comparisons with existing architectures showcase the efficiency of our model. A gated recurrent unit (GRU) architecture, albeit achieving lower error rates, incurred a computational cost 24 times larger than our proposed model. Large-scale transformer architectures, while achieving marginally lower WERs (2%–4% on LibriSpeech test-clean), require 200 times more parameters and 53,000 additional hours of training data. Modern large language models are used to improve the WERs, but require large computational resources. To further enhance performance, a small 4-g language model was integrated into our end-to-end ASR model, resulting in improved WERs. The overarching goal of this work is to provide a practical solution for practitioners dealing with limited datasets and computational resources, particularly in the context of under-resourced languages.

Test and Evaluation for verification and validation (V&V) of sensor data fusion techniques utilize methods of uncertainty analysis. The Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology identifies many attributes of metrics (i.e., semantic meaning, object metrics, and subjective quality). With the growing interest in artificial intelligence (AI) due to large data corpus access, fast compute power, and machine/deep learning (ML/DL) techniques; V&V of these methods are needed. In this paper, the enhancement of the URREF to utilize a risk assessment for decision is demonstrated towards analysis/alignment of ML/DL methods that utilize multi-modal data fusion. Evidential reasoning is considered in the use case to provide data handing reliability source and processing credibility to measure decision risk in a maritime domain awareness scenario.

ChatGPT, OpenAI’s chatbot, has gained consider-able attention since its launch in November 2022, owing to its ability to formulate articulated responses to text queries and comments relating to seemingly any conceivable subject. As impressive as the majority of interactions with ChatGPT are, this large language model has a number of acknowledged shortcomings, which in several cases, may be directly related to how ChatGPT handles uncertainty. The objective of this paper is to pave the way to formal analysis of ChatGPT uncertainty handling. To this end, the ability of the Uncertainty Representation and Reasoning Framework (URREF) ontology is assessed, to support such analysis. Elements of structured experiments for reproducible results are identified. The dataset built varies Information Criteria of Correctness, Non-specificity, Self-confidence, Relevance and Inconsistency, and the Source Criteria of Reliability, Competency and Type. ChatGPT’s answers are analyzed along Information Criteria of Correctness, Non-specificity and Self-confidence. Both generic and singular information are sequentially provided. The outcome of this preliminary study is twofold: Firstly, we validate that the experimental setup is efficient in capturing aspects of ChatGPT uncertainty handling. Secondly, we identify possible modifications to the URREF ontology that will be discussed and eventually implemented in URREF ontology Version 4.0 under development.

AI-based decision support solutions require life cycles that adequately address critical steps, such as (i) finding suitable machine learning (ML) methods for the problem at hand, (ii) preparing and executing adequate data acquisition processes and (iii) tractable evaluation of the overall solution. Understanding the data generating processes is key in achieving this. Training and test data can be seen as a result of a causal data generation process, a sampling process in which the data is collected from different sources that are influenced by multiple interdependent phenomena. This is represented by a Qualitative Model of Data Generation Processes (QM-DGP), a causal graphical model. QM-DGP facilitates analysis of the complexity of the underlying data generating processes that can inform the development of trustable ML-based solutions in multiple ways. Firstly, this analysis is the basis for the determination of the required complexity of the ML models. Secondly, it facilitates the determination of the quantities of training data supporting good learning results. Thirdly, it can provide guidance for a systematic simplification of the models, supporting tractable solutions without significantly reduced performance. The construction of QM-DGP and the analysis benefit from sound theoretical concepts, such as d-separation and I-Maps. Experimental results with simulated data indicate that the approach can be effective in predicting the required quantities of training data and the determination of the modelling complexity using different types of models.

Video data is increasingly being provided by multiple sources. These sources are used to obtain reliable feature information within the context of training data shortages. As such it becomes crucial to interpret and refine such information. Recent research on video content analysis often use deep-learning features due to their outstanding performance in different domains. These features are aggregated over time to create a video-level descriptor. In this research, we explore the potential of statistical feature aggregation methods in combining deep-learning features that represent the visual content of videos. In particular, the contributions of this paper are two-fold: Firstly, we compared statistical feature aggregation methods by selecting movie sequels, calculating their in-sequel-mean-distance and out-of-sequel-mean-distance as well as the Bhattacharyya distance between them and using Principal Component Analysis (PCA) and T-distributed Stochastic Neighbour Embedding (t-SNE) to visualise their distribution in the feature space. This is important to easily comprehend the video content without the need to watch them and understand how well the statistical feature aggregation methods represent the videos. Secondly, the performance of these aggregation methods is explored in the context of a content-based video retrieval task which is evaluated in terms of relevance. The observed results show that the statistical feature aggregation method based on variance outperforms the methods based on maximum, mean, median, median absolute deviation and interquartile range. This outcome is supported when interpreting the t-SNE visualisation method as well as Bhattacharyya distance calculations. Overall statistical feature aggregation methods which measure the spread of a distribution achieve better performance compared to methods that are a measure of location.

Tracking of multiple extended three-dimensional targets in the presence of partial occlusions from point cloud measurement sets remains a challenge but has obvious applications in autonomous vehicles. We demonstrate that existing range-image based approaches to point cloud segmentation and motion detection can be fused with fast point cloud registration techniques to aid track association. Consequently, multiple generic extended targets in point clouds can be tracked. We also show that robust occlusion handling can be achieved using temporally accumulated geometric information from co-registered point clouds in a static local reference frame. The occlusion logic is based on a novel probabilistic occlusion reasoning approach combined with a simulated geometric handling of self-occlusions. The proposed tracking pipeline was tested on synthetic target geometries superimposed on real-world backgrounds, as well as against measured data from benchmark autonomous driving datasets. The targets tracked encountered both artificially introduced and naturally occurring foreground occlusions as well as self-occlusions. Results show notable robustness to foreground occlusions. Incorrect track associations are possible when multiple targets in proximity display similar dynamics and possess nearly identical geometries. Our sequence of tracking algorithms relies on multiple tunable hyper-parameters. These parameters require further automated runtime-optimization before robust, real-time field application can be achieved.

In this paper, data from a simulated data set were fused with a significantly smaller measured data set for the purpose of training a deep neural network to identify animals from the footage of a dash cam. The trained networks were used to detect wildlife in an environment similar to game reserves in South Africa. To enable the automatic collection of data for the experiment, a simulated environment was created to simulate four classes of wildlife found in South Africa: buffalo, elephants, rhino and zebra. The network structure for the detector network selected was an adapted version of the tiny YOLOv3 network. It was discovered that using transfer learning and fine tuning resulted in two models with higher accuracy of 82.59% and 86.64% [email protected] respectively than models where no transfer learning was used. The results were achieved when tested on a testing set of digital photographic images. These networks, initialised using transfer learning, were also faster and easier to train than training using a combined data set of photographic and simulated images from scratch. The simulated environment can however not replace real-life data, as was proven by an accuracy of no better than chance for the model trained using only simulated data.

In this paper we explore automatic genre labelling of motion picture previews using audio-visual features present in movie trailers and the focus is on fusion techniques (early fusion and late fusion) and the resultant improvement on classification accuracy. This paper proposes a novel combination of deep learned features (from a pretrained VGG-16 model) obtained using a state-of-the-art shot detector and hand-crafted audio features. This combination of features and an associated comparison of early and late fusion with these features has not been attempted in the literature before. Furthermore, two popular fusion techniques and three distinct classification algorithms are investigated to determine the optimal fusion technique and classifier combination. The study uses a subset of the LMTD-9 movie trailer dataset with selected genres (action, comedy, drama and horror). The best performing low-level audio features are comprised of timbre features extracted using the MIRtoolbox followed by standalone mel-frequency cepstral coefficients. The best performing high-level audio feature is tonality. Audio features are augmented by visual features extracted using a pre-trained convolutional neural network (VGG-16). Feature fusion (early and late fusion) methods are investigated together with classification methods such as extreme gradient boosting, support vector machine and a neural network. Evaluation metrics such as precision, recall, confusion matrices and F1 score are used to measure classification accuracy. Early fusion methods outperform late fusion methods with a classification performance gain of approximately 10% for a four class classification problem. The best classification performance for early fusion obtained with a support vector machine is (73.12% accuracy), followed by the extreme gradient boosting classifier (69.37% accuracy) and neural network classifier (67.50% accuracy), whereas chance is 25%. It is shown that superior classification performance can be achieved by employing early feature fusion of low-level audio descriptors, high-level audio descriptors and high-level visual feature descriptors together with suitable classifiers.

Systems for threat analysis enable users to understand the nature and behavior of threats and to undertake a deeper analysis for detailed exploration of threat profile and risk estimation. Models for threat analysis require significant resources to be developed and are often relevant to limited application tasks. This paper investigated the implicit and explicit uncertainty assessments to be taken into account for threat analysis systems to be effective for providing a relevant threat characterization. The intent of this paper is twofold. The first is to present and discuss an approach to define a model for cyber threats within a simplified expert model and to translate it into a Bayesian network as a tool for the development of practical scenarios for cyber threats analysis. The second is to address the question of assessing the Bayesian network build and its intrinsic knowledge representation model and to show how modeling decisions impact the outcome of the system. The paper describes the construction of an expert model and the corresponding BN to analyze cyber threats, investigates various types of induced uncertainty with the URREF criteria simplicity and expressiveness and implements an assessment procedure to evaluate the overall approach.

Sensormanagement is an important function of any data fusion center as the output of a fusion system is dependent on the quality of the information collected. In this paper, the scheduling aspect of sensor management function is implemented using Response Surface Modeling (RSM). Applying RSM requires formulating the sensor management function as an objective function. The benefit of RSM over prior global optimization approaches is the simplification of the evaluation of this objective function to find global optima. This leads to either reduced computational requirements and/ or shorter due times for creating sensor schedules. This work shows the utility of RSM towards scheduling multiple sensors, and seeks to introduce RSM to the sensor management community. It is shown that the RSM scheduler provides a significant improvement towards reducing the number of missed targets in a surveillance radar network. This is compared to performing a uniform scanning regime (or sequential stepped scan) often employed. Very few iterations are required to provide this gain. The RSM technique also quickly determines where the most effective use of sensor resources needs to be applied. Consequently, it spends more radar dwell time on these beam locations.

This paper presents a Bayesian tracking approach that exploits various types of context information. The filtering accuracy and precision are improved by using uncertain information about (i) the constraints on the target mobility, (ii) environmental influences on the sensor performance and (iii) typical target behaviors. The approach combines particle filters with exact Bayesian networks. The overall process is equivalent to approximate inference on elaborate dynamic Bayesian networks that systematically capture non-trivial correlations between the estimated states of the dynamic processes, the associated observations and the various factors influencing the dynamic processes. The particle filter supports reasoning about continuous dynamic processes spanning large areas, while the Bayesian networks are used for the implementation of advanced sensor models and for the fusion of uncertain data on mobility constraints. The derivation of the solution is based on the decomposability principles of Bayesian networks. The approach is illustrated with the help of a challenging wildlife protection application. A set of qualitative experiments shows the improvement in tracking performance by considering the different types of context information.

The choices of the uncertainty representations and reasoning methods have a critical impact on the development and deployment of modern fusion solutions. They influence the development effort, the quality of the resulting solutions as well as the deployment costs. However, such choices require an analysis that considers many operational and theoretical aspects. The Uncertainty Representation and Reasoning Evaluation Framework (URREF) concepts enable such an analysis. The proposed URREF -driven development approach establishes relations between the URREF criteria and evaluation subjects in the context of a development and deployment life cycle. In this way the relevant theoretical elements of the fusion techniques employed are emphasized and evaluated at various stages of the development process, facilitating informed design choices as well as systematic and tractable evaluation of complex fusion solutions. The concepts are illustrated through the assessment of a high-level fusion approach supporting estimation of the whereabouts of wildlife poachers.

The International Society of Information Fusion (ISIF) Evaluation Techniques for Uncertainty Representation Working Group (ETURWG) investigates the quantification and evaluation of all types of uncertainty regarding the inputs, reasoning and outputs of the information fusion process. The ETURWG is developing an Uncertainty Representation and Reasoning Framework (URREF) ontology for this purpose. This paper outlines a start towards the process of defining metrics for the URREF data criteria, which will align the URREF ontology with practical application. A criterion can be evaluated according to several metrics, and a metric can be applied to several criteria. As such, the ontology would have to reflect the nature of a many-to-many mapping between criteria and metrics. The main findings and suggestions of the paper advancing the use of URREF are: 1) The Weight of Information (WoI) is dependent on data criteria, which in turn depend on source criteria. 2) Criteria and metrics that apply to evidence (typically an input of the fusion system), could equally apply to the fusion system outputs or internal information, which in turn could form the inputs of another system. As such the word “Evidence” in the terms “Piece of Evidence” and “Weight of Evidence” should be replaced by the word “Information”. 3) Accuracy and precision and associated metrics are ubiquitous in the URREF ontology and can evaluate many parts of the fusion system. 4) The weight of information also assumes an important position in the ontology, as it depends on several source and data criteria.

The question addressed in this paper is “what” is to be evaluated by the Uncertainty Representation and Reasoning Evaluation Framework (URREF) ontology. We thus identify the elements composing uncertainty representation and reasoning approaches, which constitute various subjects being assessed. We distinguish between primary evaluation subjects (Uncertainty Representation and Reasoning components of the fusion algorithm), and secondary evaluation subjects (source of information, piece of information, fusion method and mathematical model). This paper proposes a list of source quality criteria to be added to the ontology and establishes formal links between the secondary and primary evaluation subjects. The key contribution of the paper is the update of the definitions of sub-criteria of the Expressiveness criterion together with suggestions for complementary concepts to be included in the ontology (type of scale, type of uncertainty expression). Conclusions are drawn to extend the work in using the expressiveness criterion for information fusion analysis.