This paper introduces QCaption, a novel video captioning and Q&A pipeline that enhances video analytics by fusing three models: key frame extraction, a Large Multimodal Model (LMM) for image-text analysis, and a Large Language Model (LLM) for text analysis. This approach enables integrated analysis of text, images, and video, achieving performance improvements over existing video captioning and Q&A models; all while remaining fully self-contained, adept for on-premises deployment. Experimental results using QCaption demonstrated up to $\mathbf{4 4. 2 \%}$ and $\mathbf{4 8. 9 \%}$ improvements in video captioning and Q&A tasks, respectively. Ablation studies were also performed to assess the role of LLM on the fusion on the results. Moreover, the paper proposes and evaluates additional video captioning approaches, benchmarking them against QCaption and existing methodologies. QCaption demonstrate the potential of adopting a model fusion approach in advancing video analytics.

Performing crowd management in large scale outdoor events at night is a challenging yet essential task for public safety and security purposes. Traditional methods of carrying out crowd counting require the deployment of massive manpower and are unable to provide a reliable count for effective resource and manpower planning. In recent years, deep learning based crowd counting methods trained on static images were introduced. However, there still exist real world challenges of variations in crowd density across the scene, illumination, environmental conditions, and perspective problems which these methods are unable to fully address. This paper attempts to address the problems of varying crowd densities and illumination through a crowd counting pipeline that fuses illumination enhancement processes with crowd density estimation and crowd localisation techniques to achieve improved accuracy for crowd counts from live UAV video feeds. This data fusion pipeline approach has been demonstrated to provide improved count accuracy on both dataset and real-world images compared against standalone state-of-the-art methods.

In this paper, we develop an approach for measurement-to-track association (M2TA) in the presence of (unknown) camera panning and zooming from drone-captured video. Standard M2TA methods assume that the target motion can be used to predict the “measurement association regions” for the bounding boxes. However, if there is a sudden state change due to camera shift (panning) and zooming, it will lead to incorrect associations and poor tracking results. To solve this, the zoom ratio and panning in 2D coordinates are used to describe the camera motion parameters in each frame. The estimated parameters are obtained by a grid search combined with global assignment or directly solved using the linear least squares method, which is also combined iteratively with assignment. The goal is to achieve correct M2TA by adjusting the predicted measurements using the estimated camera parameters. These “improved” predictions can also be used to update the target state with filtering algorithms. Frames with panning or/and zooming from real data are used to illustrate the effectiveness of the proposed methods and compared with the validation gate method based on inflated covariances.

Feature fusion is important for providing enhancements of data authenticity in both traditional and deep learning pattern analysis. Classical serial fusion concatenates multiple feature sets followed by dimensionality reduction using principal component analysis (PCA), linear discriminant analysis (LDA), canonical correlation analysis (CCA) etc. CCA-based feature fusion is a main technique for exploring the mutual relationships of multiple feature sets. It considers the correlation of multiple feature sets during dimensionality reduction. In traditional CCA-based feature fusion and extraction, the natural groupings of features are directly used. It is still unclear whether the natural groupings of features are optimal for CCA-based fusion. In this paper, we propose a feature regrouping algorithm for CCA-based feature fusion and extraction through normalized cut (FR-NC). Feature correlation analysis is incorporated into normalized cut, in which the intra-group correlation is maximized, and the extra-group correlation is minimized simultaneously. CCA-based feature fusion is performed on the regrouped features. The proposed feature regrouping algorithm aims to provide enhanced fused features for pattern classification. Extensive experiments have proved its effectiveness.

This paper considers the problem of target tracking using an asynchronous multistatic system with unknown transmitter positions. In such a system, the receiver is considered as the own sensor to perform passive tracking. It listens to the signals from at least two non-cooperative transmitters via direct and indirect (bouncing from targets) paths. The transmitters and targets are then tracked based on the measured bearings and the bistatic ranges (derived from the TDOA of the direct and indirect path signals). Since the transmitter positions are unknown, they have to be estimated, and their estimates will contain errors. To cope with these errors, we develop an iterated least squares estimator with covariance inflation (ILS-CI) for track initiation, and apply the covariance inflation filter (CIF) for track update. Four approaches, namely the optimal, simple, covariance inflation (CI) and combined approaches, with different strategies in track initiation and track update, are proposed to solve this tracking problem. Their performances are evaluated through simulation tests.

With the exponential growth of web meta-data, exploiting multimodal online sources via standard search engine has become a trend in visual recognition as it effectively alleviates the shortage of training data. However, the web meta-data such as text data is usually not as cooperative as expected due to its unstructured nature. To address this problem, this paper investigates the numerical representation of web text data. We firstly adopt convolutional neural network (CNN) for web text modeling on top of word vectors. Combined with CNN for image, we present a multimodal fusion to maximize the discriminative power of visual and textual modality data for decision level and feature level simultaneously. Experimental results show that the proposed framework achieves significant improvement in large-scale image classification on Pascal VOC-2007 and VOC-2012 datasets.

Information fusion aims to exploit truthful knowledge from various sources in a reliable and accurate way. Fusion of information can be conducted at three abstraction levels including feature level, score level and decision level. The feature fusion approaches have the advantages of preserving effective discriminative structure underlying various features. In this paper, we propose an effective feature fusion algorithm based on intra-class and extra-class discriminative correlation analysis (IEDCA), aiming to eliminate between-class correlation and retain enough feature dimension for correlation analysis. IEDCA explores the intra-class correlation including both the pairs-wise correlation like CCA-based feature fusion approaches and the correlation across different features within the same class. Our proposed method can be used in unimodal feature fusion as well as multimodal feature fusion, and extensive experiments have proved its effectiveness.

Acoustic propagation delay has not been investigated for a continuous wave multistatic sonar tracking system except for the recent study conducted by Jauffret et al. [4], which estimates the trajectory of a constant velocity target. The results showed that the estimate bias caused by the propagation delay is not negligible, especially for a bistatic system. This paper develops an interacting multiple model unscented Gauss-Helmert filter with numerical Jacobian (IMM-UGHF-NJ) to track a maneuvering target with propagation delay using a bistatic sonar system. The IMM-UGHF-NJ can overcome the two tracking challenges introduced by the delay, namely, implicit state transition model and lack of analytical expression of the Doppler shifted frequency in the measurement model. Simulation tests have been conducted, and the results show that the IMM-UGHF-NJ can reduce the estimation error significantly, especially for fast moving targets.

The Interacting Multiple Model (IMM) algorithm is a widely accepted state estimation scheme for solving maneuvering target tracking problems, which are generally nonlinear. During the IMM filtering process, serious errors can arise when a Gaussian mixture of posterior probability density functions is approximated by a single Gaussian. Particle filters (PFs) are effective in dealing with nonlinearity and non-Gaussianity. This work considers an IMM algorithm that includes a constant velocity model, a constant acceleration model and a 3D turning rate (3DTR) model for tracking three-dimensional (3D) target motion, using various combinations of nonlinear filters. In existing literature on combining IMM and particle filtering techniques to tackle difficult target maneuvers, a PF is usually used in every model. In comparison, simulation results show that by using a computationally economical PF in the 3DTR model and Kalman filters in the remaining models, superior performance can be achieved with significant reduction in computational costs.

Intent inference involves analyzing the actions and activities of a target of interest to deduce its purpose. In an environment cluttered with many targets, loaded with information, and under stress, the human may not be able to perform well. Hence a cognitive aid that could derive possible intent inference and monitor the target may help augment human cognition and assist critical human decision making. This paper reports research on two applications: determining the likelihood of weapon delivery by an attack aircraft under military surveillance and conformance monitoring in air traffic control systems. The proposed solution is based on flight profile analysis. Simulation process comprises Interacting Multiple Model-based state estimation and Mamdani-type fuzzy inference to deduce possibilities of weapon delivery and of nonconforming aircraft behavior respectively. Results verify that the method is feasible and provides timely inference that will aid human cognition and hence assist decision making.

In the area of Process Refinement under Level 4 Data Fusion of the JDL model [1,2], high-level sensor management is often performed by human operators manning sensor systems who constantly have to monitor the situational and sensor picture for critical events and dynamically employ myriad sensors’ functions to carry out mission-specific tasks. To assist the human operators in dealing better with the intense pressure to perform effectively in such environments, adaptive knowledge bases capable of capturing human operators’0339 behavioural patterns can be harnessed to augment the task-oriented decision making process of sensor management. However, the unique problem domain in which human operators exercise sensor management functions has direct impact on obtainable training data and imposes several performance requirements on the adaptive knowledge bases. Several rule-learning algorithms [4-7] do not readily fulfil the identified requirements and selecting a more suitable alternative constitutes the focus of this paper. This paper selects the adaptive online-learning Evolving Fuzzy Neural Network (EFUNN) [8,9] and details two algorithmic and one qualitative contribution that enhance EFUNN’s ability to realize the construction of adaptive knowledge bases. The two algorithmic contributions consist of modifications of EFUNN’s original learning mechanism to handle training records with outlying inputs and those with contradictory class outputs that characterise the obtainable training data. The qualitative contribution suggests how multiple EFUNNs can be mapped to respective task-oriented rule-sets giving rise to adaptive knowledge bases that assist the human operators in selecting the right observation tasks.

This paper will specifically undertake the task of improving the passive sonar system using Self- Organizing Map. Localizing multiple targets is a challenging problem as passive sonar sensors are only able to detect the targets’ bearing angle. An effective way to find the targets location is by triangulation. However, in multi-sensor multi-target environment, ghost targets are introduced during the triangulation process. Self-Organizing Map based on neural network is one of the most recently used methods proposed to extract the true targets. This paper will introduce two improvements to the Self-Organizing Map. The first improvement is to initialize the neurons based on the preliminary triangulation point’s distribution. This results in a faster first-time-seen of the targets. The second improvement is to apply the assumption that each bearing line is associated with only one target. This results in the reduction of the amount of false tracks detected.

In this paper, we present two novel methods to handle the fusion of multiple Bayesian Network knowledge fragments which we termed N-Combinator and N-Clone. In DSO National Laboratories, we have developed a cognition based dynamic reasoning machine called D’Brain capable of performing high level data fusion. Knowledge is encapsulated in D’Brain as Bayesian Networks knowledge fragments. D’Brain is dynamic in its reasoning mechanism that resembles human reasoning, where the knowledge structure is ever evolving with the different sources of observable inputs. N-Combinator and N-Clone are the methods used in the dynamic reasoning mechanism. Experiments have shown the good performance of these two methods.

With growing emphasis on research and applications in data fusion, it is important to know how to measure fusion system performance. While much of the current research is involved with evaluating fusion systems based on the quality of their outputs, this paper describes the expansion of this scope to include the complexity of the inputs. Since the quality of the outputs is highly dependent on the difficulty of the input scenario, the methodology described in this paper provides us with a more well-rounded interpretation of fusion system performance. Moreover, characterization and quantification of input scenarios allows for the creation and variation of test scenarios, leading to more efficient optimization of fusion systems. A series of complexity and performance metrics were designed according to a common standard to measure the various aspects of the input scenario and the fusion system solution. Finally, these metrics are combined to obtain an assessment index which provides a description of the fusion system performance

This paper presents a set of novel and efficient algorithms for fusion of emitters from multiple sensors, where a signal from one sensor system could either be a bearing line or a fix. This comprehensive solution includes fusion algorithms for multiple fixes, multiple bearing lines, and fix with bearing. Testing using a high-fidelity simulation system demonstrated that the algorithms are able to reduce the location error significantly