This article examines performance evaluation criteria for basic vision tasks involving sets of objects namely, object detection, instance-level segmentation and multi-object tracking. The rankings of algorithms by a criterion can fluctuate with different choices of parameters, e.g. Intersection over Union (IoU) threshold, making their evaluations unreliable. More importantly, there is no means to verify whether we can trust the evaluations of a criterion. This work suggests a notion of trustworthiness for performance criteria, which requires (i) robustness to parameters for reliability, (ii) contextual meaningfulness in sanity tests, and (iii) consistency with mathematical requirements such as the metric properties. We observe that these requirements were overlooked by many widely-used criteria, and explore alternative criteria using metrics for sets of shapes. We also assess all these criteria based on the suggested requirements for trustworthiness.

This paper proposes an online multi-camera multi-object tracker that only requires monocular detector training, independent of the multi-camera configurations, allowing seamless extension/deletion of cameras without retraining effort. The proposed algorithm has a linear complexity in the total number of detections across the cameras, and hence scales gracefully with the number of cameras. It operates in the 3D world frame, and provides 3D trajectory estimates of the objects. The key innovation is a high fidelity yet tractable 3D occlusion model, amenable to optimal Bayesian multi-view multi-object filtering, which seamlessly integrates, into a single Bayesian recursion, the sub-tasks of track management, state estimation, clutter rejection, and occlusion/misdetection handling. The proposed algorithm is evaluated on the latest WILDTRACKS dataset, and demonstrated to work in very crowded scenes on a new dataset.

Meeting or conference assistance is a popular application that typically requires compact configurations of co-located audio and visual sensors. This paper proposes a novel solution for online separation of an unknown and time-varying number of moving sources using only a single microphone array co-located with a single visual device. The approach exploits the complementary nature of simultaneous audio and visual measurements, accomplished by a model-centric 3-stage process of detection, tracking, and (spatial) filtering, which performs separation in a block-wise or recursive fashion. Fusing the measurements requires solving the multi-modal space-time permutation problem, since the audio and visual measurements reside in different observation spaces, but also are unidentified or unlabeled (with respect to the unknown and time-varying number of sources), and are subject to noise, extraneous measurements and missing measurements. A labeled random finite set tracking filter is applied to resolve the permutation problem and recursively estimate the source identities and trajectories. A time-varying set of generalized side-lobe cancellers is constructed based on the tracking estimates to perform online separation. Evaluations are undertaken with live human speakers.

This paper proposes a novel solution for separating an unknown and time-varying number of moving acoustic sources in a blind setting using multiple microphone arrays. A standard steered-response power phase transform method is applied to extract source position measurements, which inevitably contain noise, false detections, missed detections, and are not labeled with the source identities. The imperfect measurements lead to the space-time permutation problem, as there is no information on how the measurements are associated to the sources in space, nor how the measurements are connected across time, if at all. To solve this problem, a labeled random finite set tracking framework is adopted to jointly estimate the source positions and their labels or identities. Based on these trajectory estimates, a corresponding set of time-varying generalized side-lobe cancellers is constructed to perform source separation. The overall algorithm operates in a block-wise or an online fashion and is scalable with the number of microphone arrays. The quality of the measurements, tracking, and separation, are evaluated respectively, with the OSPA metric, OSPA(2)metric, and ITU-T P.835 based listening tests, on both real-world and simulated data.

The traditional method of applying the optimal subpattern assignment (OSPA) metric cannot fully evaluate multitarget tracking performance, as it does not account for phenomena such as track label switching, and track fragmentation. The OSPA(2)has been proposed as a technique for applying the OSPA distance in a way that captures these effects, while retaining the properties of a true metric. In this paper, we demonstrate the behaviour of the OSPA(2)on some numerical examples, discuss some of its advantages and limitations, and show that it is capable of being applied to performance evaluation of large-scale scenarios in the order of a thousand targets.

This paper extends the generalized labeled multi-Bernoulli (GLMB) tracking filter to a batch multi-target tracker. In a labeled random finite set formulation, a multi-target tracking filter propagates the labeled multi-target filtering density while a batch multi-target tracker propagates the labeled multi-target posterior density. The GLMB filter is an analytic solution to the labeled multi-target filtering recursion. In this work, we show that the GLMB filter can be extended to an analytic multi-object posterior recursion.

This article proposes a framework for model-based point pattern learning using point process theory. Likelihood functions for point pattern data derived from point process theory enable principled yet conceptually transparent extensions of learning tasks, such as classification, novelty detection and clustering, to point pattern data. Furthermore, tractable point pattern models as well as solutions for learning and decision making from point pattern data are developed.

This letter proposes analytical expressions for the fusion of certain classes of labeled multiobject densities via Kullback-Leibler averaging. Specifically, we provide analytical fusion rules for the labeled multi-Bernoulli and marginalized δ-generalized labeled multi-Bernoulli families of labeled multiobject densities. Information fusion via Kullback-Leibler averaging ensures immunity to double counting of information and is essential to the development of effective multiagent multiobject estimation.

This paper proposes an efficient implementation of the multi-sensor generalized labeled multi-Bernoulli (GLMB) filter. The solution exploits the GLMB joint prediction and update together with a new technique for truncating the GLMB filtering density based on Gibbs sampling. The resulting algorithm has a complexity in the order of the product of the number of measurements from each sensor, and quadratic in the number of hypothesized objects.

Clustering is one of the most common unsupervised learning tasks in machine learning and data mining. Clustering algorithms have been used in a plethora of applications across several scientific fields. However, there has been limited research in the clustering of point patterns - sets or multi-sets of unordered elements - that are found in numerous applications and data sources. In this paper, we propose two approaches for clustering point patterns. The first is a non-parametric method based on novel distances for sets. The second is a model-based approach, formulated via random finite set theory, and solved by the Expectation-Maximization algorithm. Numerical experiments show that the proposed methods perform well on both simulated and real data.

Point patterns are sets or multi-sets of unordered elements that can be found in numerous data sources. However, in data analysis tasks such as classification and novelty detection, appropriate statistical models for point pattern data have not received much attention. This paper proposes the modelling of point pattern data via random finite sets (RFS). In particular, we propose appropriate likelihood functions, and a maximum likelihood estimator for learning a tractable family of RFS models. In novelty detection, we propose novel ranking functions based on RFS models, which substantially improve performance.

In this paper, we extend the notion of Cauchy-Schwarz divergence to point processes and establish that the Cauchy-Schwarz divergence between the probability densities of two Poisson point processes is half the squared L2-distance between their intensity functions. Extension of this result to mixtures of Poisson point processes and, in the case where the intensity functions are Gaussian mixtures, closed form expressions for the Cauchy-Schwarz divergence are presented. Our result also implies that the Bhattacharyya distance between the probability distributions of two Poisson point processes is equal to the square of the Hellinger distance between their intensity measures. We illustrate the result via a sensor management application where the system states are modeled as point processes.

Quantitative analysis of the dynamics of tiny cellular and sub-cellular structures, known as particles, in time-lapse cell microscopy sequences requires the development of a reliable multi-target tracking method capable of tracking numerous similar targets in the presence of high levels of noise, high target density, complex motion patterns and intricate interactions. In this paper, we propose a framework for tracking these structures based on the random finite set Bayesian filtering framework. We focus on challenging biological applications where image characteristics such as noise and background intensity change during the acquisition process. Under these conditions, detection methods usually fail to detect all particles and are often followed by missed detections and many spurious measurements with unknown and time-varying rates. To deal with this, we propose a bootstrap filter composed of an estimator and a tracker. The estimator adaptively estimates the required meta parameters for the tracker such as clutter rate and the detection probability of the targets, while the tracker estimates the state of the targets. Our results show that the proposed approach can outperform state-of-the-art particle trackers on both synthetic and real data in this regime.

Track-before-detect (TBD) refers to an alternative approach to tracking which utilizes the full sensor information rather than detections obtained from thresholding. In this letter we investigate whether forward-backward smoothing for TBD can increase performance. We propose a novel algorithm based on the random finite set framework which incorporates the TBD sensor model with multi-scan information. The algorithm is tested on a typical scenario which confirms improved tracking.

It was recently demonstrated that the Gaussian Mixture Cardinalised Probability Hypothesis Density (GMCPHD) filter can be used when the clutter density is unknown. Here we examine the performance of this filter, and as one would expect, it does not do as well as the conventional GMCPHD with matched clutter density. To improve the performance, we propose a bootstrap filtering scheme, and demonstrate by simulations on a bearings-only multitarget filtering scenario, that it is capable of performing almost as well as the matched GMCPHD filter.

A new method is presented for integration of audio and visual information in multiple target tracking applications. The proposed approach uses a Bayesian filtering formulation and exploits multi-Bernoulli random finite set approximations. The work presented in this paper is the first principled Bayesian estimation approach to solve the sensor fusion problems that involve intermittent sensory data (e.g. audio data for a person who occasionally speaks.) We have examined our method with case studies from the SPEVI database. The results show nearly perfect tracking of people not only when they are silent but also when they are not visible to the camera (but speaking).

This paper proposes an integrated Bayesian frame work for feature-based simultaneous localization and map building (SLAM) in the general case of uncertain feature number and data association. By modeling the measurements and feature map as random finite sets (RFSs), a formulation of the feature-based SLAM problem is presented that jointly estimates the number and location of the features, as well as the vehicle trajectory. More concisely, the joint posterior distribution of the set-valued map and vehicle trajectory is propagated forward in time as measurements arrive, thereby incorporating both data association and feature management into a single recursion. Furthermore, the Bayes optimality of the proposed approach is established. A first-order solution, which is coined as the probability hypothesis density (PHD) SLAM filter, is derived, which jointly propagates the posterior PHD of the map and the posterior distribution of the vehicle trajectory. A Rao-Blackwellized (RB) implementation of the PHD-SLAM filter is proposed based on the Gaussian-mixture PHD filter (for the map) and a particle filter (for the vehicle trajectory). Simulated and experimental results demonstrate the merits of the proposed approach, particularly in situations of high clutter and data association ambiguity.

Almost every single-view visual multi-target tracking method presented in the literature includes a detection routine that maps the image data to point measurements relevant to the target states. These measurements are commonly further processed by a filter to estimate the number of targets and their states. This paper presents a novel visual tracking technique based on a multi-object filtering algorithm that operates directly on the image observations without the need for any detection. Experimental results on tracking sport players show that our proposed method can automatically track numerous interacting targets and quickly finds players entering or leaving the scene.

The probability hypothesis density (PHD) recursion is a first moment approximation to the multi-target Bayes filter which propagates the posterior intensity of the random finite set of targets in time. The cardinalized PHD (CPHD) recursion is a generalization of the PHD recursion, which jointly propagates the posterior intensity and the posterior cardinality distribution. Using the recently developed closed form solutions to both the PHD and CPHD recursions for linear Gaussian multi-target models, we present a comparative study of their performances