Existing trackers based on Poisson measurement process often struggle with efficiency and accuracy in large-scale tracking under heavy clutter. To overcome this, we introduce PiVoT, a scalable, robust multi-object tracker capable of efficiently detecting and tracking a large, varying number of objects, along with their shapes, existence probabilities, and measurement rates, even in heavy clutter. PiVoT employs a novel two-stage variational inference routine to achieve inference tractability and closed-form, parallelisable updates. Efficiency is further enhanced by early identification and removal of ineffective birth objects and designing highly simplified, much faster, yet equivalent variational updates. Additionally, PiVoT inherently offers efficient clutter-robust clustering, an innovation that can also enhance existing trackers that depend on supplementary clustering techniques. Experiments demonstrate PiVoT's clear accuracy and efficiency gains over existing methods, while also highlighting its ability to track a thousand closely spaced objects in under a second on a standard laptop without gating.

Integrated sensing and communication (ISAC) is set to transform data transmission and real-time sensing in sixth-generation (6G) networks. By offering high capacity, free-space optical (FSO) links can play a vital role as a backhaul solution, complementing radio frequency (RF) technologies such as millimeter wave (mmWave) to enhance 6G reliability. While adverse weather conditions such as fog can reduce the reliability of FSO links due to atmospheric attenuation, the back-scattered light it generates may also enable real-time sensing of atmospheric channel gain at the transmitter. This paper proposes a novel user offloading scheme utilizing an optical ISAC (O-ISAC) framework within a single-cell network, supported by a ground base station (GBS) and enhanced by an unmanned aerial vehicle (UAV). The UAV connects to the gateway via an FSO backhaul link, estimating the channel gain based on back-scattered light and dynamically optimizing its position to efficiently offload users in the downlink. Numerical results demonstrate the UAV’s effectiveness in optimizing its position under various weather conditions and show that the proposed deployment scheme outperforms existing UAV offloading methods. The framework is evaluated using a real dataset of hourly visibility measurements in Edinburgh, underscoring the significance of optical channel sensing for overall system performance and providing insights into how sensing impacts communication efficiency.

Integrated sensing and communication (ISAC) is expected to transform data transmission and real-time sensing, enhancing sixth-generation (6G) networks. Free-space optical (FSO) communication is a key 6G backhaul solution, complementing radio frequency (RF) technologies like millimeter wave (mmWave) for improved network reliability. However, adverse weather can significantly reduce FSO link reliability due to atmospheric attenuation. Such adverse weather conditions also increase the level of back-scattered light, potentially enabling the real-time sensing of the atmospheric channel gain at the transmitter side. Therefore, this paper proposes a novel optical ISAC (O-ISAC) framework, where the back-scattered light from the FSO communication signal is used as the sensing feedback signal. This O-ISAC framework is analyzed considering a single-cell network aided by an unmanned aerial vehicle (UAV) to support edge users. The UAV is connected to the gateway via a FSO backhaul link while estimating the FSO channel gain based on the back-scattered light and dynamically optimizing its trajectory. The aim of this adaptive O-ISAC system is to maximize the end-to-end network throughput of the edge users while considering FSO backhaul capacity and the UAV's directional antenna beamwidth and bandwidth allocation. Numerical results demonstrate that UAV can effectively optimize its trajectory by adjusting the antenna beamwidth and downlink bandwidth allocation at different weather conditions. The proposed framework is tested using hourly visibility data from Edinburgh, demonstrating that optical channel sensing is crucial for the system's overall performance.

This paper explores the increasing demand for accurate and resilient multi-sensor fusion techniques, particularly within 3D tracking systems enhanced by drone technology. Employing the adaptive kernel Kalman filter (AKKF) methodology within the Stone Soup framework, our research seeks to develop robust fusion approaches capable of seamlessly amalgamating data from a multi-sensor arrangement with fixed ground sensors and dynamic sensors mounted on drones. By capitalising on the adaptive nature of the $A K K F$, we aim to refine the precision and dependability of 3D object tracking in intricate scenarios. Through empirical evaluations, we illustrate the effectiveness of our proposed AKKF-based fusion strategies in enhancing tracking performance within the Stone Soup framework, thus contributing to the advancement of multi-sensor fusion methodologies within this framework.

In the context of sixth generation (6G) and beyond, millimeter wave (MMW) is a vital technology for high data-rate applications in the realm of radio frequency (RF) communications due to its high bandwidth and low interference. However, adverse weather conditions, particularly rain, can significantly impact the MMW link capacity. In this study, we propose to exploit unmanned aerial vehicles (UAVs) as hybrid relays to improve the availability of MMW links in adverse weather conditions. The UAV relays exploit an intermediate free-space optical (FSO) relay link without impacting the RF source and destination nodes. We maximize the end-to-end channel capacity by optimizing the UAVs’ locations based on the negatively correlated MMW and FSO link coverage lengths. We compare the proposed model’s performance with benchmark systems, including a single MMW link, a static location relay-aided MMW system, and static hybrid FSO-RF systems. The numerical results clearly demonstrate the proposed scheme’s superiority, including better performance when tested on realistic rainy channel conditions based on measured weather information in the United Kingdom.

Pneumonia, a respiratory disease often caused by bacterial infection in the distal lung, requires rapid and accurate identification, especially in settings such as critical care. Initiating or de-escalating antimicrobials should ideally be guided by the quantification of pathogenic bacteria for effective treatment. Optical endomicroscopy is an emerging technology with the potential to expedite bacterial detection in the distal lung by enabling in vivo and in situ optical tissue characterisation. With advancements in detector technology, Optical endomicroscopy can utilize fluorescence lifetime imaging (FLIM) to help detect events that were previously challenging or impossible to identify using fluorescence intensity imaging. In this paper, we propose an iterative Bayesian approach for bacterial detection in FLIM. We model the FLIM image as a linear combination of background intensity, Gaussian noise, and additive outliers (labelled bacteria). While previous bacteria detection methods model anomalous pixels as bacteria, here the FLIM outliers are modelled as circularly symmetric Gaussian-shaped objects, based on their discrete shape observed through visual analysis and the physical nature of the imaging modality. A Hierarchical Bayesian model is used to solve the bacterial detection problem where prior distributions are assigned to unknown parameters. A Metropolis-Hastings within Gibbs sampler draws samples from the posterior distribution. The proposed method’s detection performance is initially measured using synthetic images, and shows significant improvement over existing approaches. Further analysis is conducted on real Optical endomicroscopy FLIM images annotated by trained personnel. The experiments show the proposed approach outperforms existing methods by a margin of +16.85% (F1) for detection accuracy.

Abstract In this paper, we introduce our unique dataset of fluorescence lifetime imaging endo/microscopy (FLIM), containing over 100,000 different FLIM images collected from 18 pairs of cancer/non-cancer human lung tissues of 18 patients by our custom fibre-based FLIM system. The aim of providing this dataset is that more researchers from relevant fields can push forward this particular area of research. Afterwards, we describe the best practice of image post-processing suitable per the dataset. In addition, we propose a novel hierarchically aggregated multi-scale architecture to improve the binary classification performance of classic CNNs. The proposed model integrates the advantages of multi-scale feature extraction at different levels, where layer-wise global information is aggregated with branch-wise local information. We integrate the proposal, namely ResNetZ, into ResNet, and appraise it on the FLIM dataset. Since ResNetZ can be configured with a shortcut connection and the aggregations by Addition or Concatenation, we first evaluate the impact of different configurations on the performance. We thoroughly examine various ResNetZ variants to demonstrate the superiority. We also compare our model with a feature-level multi-scale model to illustrate the advantages and disadvantages of multi-scale architectures at different levels.

This letter incorporates the adaptive kernel Kalman filter (AKKF) into the belief propagation (BP) algorithm for multi-target tracking (MTT) in single-sensor systems. The algorithm is capable of tracking an unknown and time-varying number of targets, in the presence of false alarms, clutter and measurement-to-target association uncertainty. Experiment results reveal that the proposed method has a favourable tracking performance using the generalized optimal sub-patten assignment (GOSAP) metrics at substantially less computation cost than the particle filter (PF) based multi-target tracking (MTT) BP algorithm.

Convolutional Neural Networks (CNNs) are a popular choice for estimating Direction of Arrival (DoA) without explicitly estimating delays between multiple microphones. The CNN method first optimises unknown filter weights (of a CNN) by using observations and ground-truth directional information. This trained CNN is then used to predict incident directions given test observations. Most existing methods train using spectrally-flat random signals and test using speech. In this paper, which focuses on single source DoA estimation, we find that training with speech or music signals produces a relative improvement in DoA accuracy for a variety of audio classes across 16 acoustic conditions and 9 DoAs, amounting to an average improvement of around 17% and 19% respectively when compared to training with spectrally flat random signals. This improvement is also observed in scenarios in which the speech and music signals are synthesised using, for example, a Generative Adversarial Network (GAN). When the acoustic environments during test and training are similar and reverberant, training a CNN with speech outperforms Generalized Cross Correlation (GCC) methods by about 125%. When the test conditions are different, a CNN performs comparably. This paper takes a step towards answering open questions in the literature regarding the nature of the signals used during training, as well as the amount of data required for estimating DoA using CNNs.

Mobile Cloud Computing or Fog computing refers to offloading computationally intensive algorithms from a mobile device to the cloud or an intermediate cloud in order to save resources e.g., time and energy in the mobile device. This paper proposes new solutions for situations when the cloud or fog is not available. First, the sensor network is modelled using a network of queues, then a linear programming technique is used to make scheduling decisions. Various centralized and distributed algorithms are then proposed, which improves overall system performance. Extensive simulations show slightly higher energy usage in comparison to the baseline non-offloading case, however, the job completion rate is significantly improved, the efficiency score metric shows the extra energy usage is justified. The algorithms have been simulated in various environments including high and low bandwidth, partial connectivity, and different rate of information exchanges to study the pros and cons of the proposed algorithms.

This paper proposes a method for minimum mean squared error (MMSE) adaptive waveform design (AWD) in multiple-input-multiple-output (MIMO) active sensing systems which are used to track moving targets. The method proposed herein prompts two computational improvements compared to a related method for static targets. Consideration of moving targets also introduces the possibility of `model mismatch' between the actual motion of the targets, and the model available to the MMSE AWD system. Results show that the proposed method leads to an improvement in mean squared error performance of up to 29% compared to the non-adaptive case.

Parametric Bayesian spectral estimation methods have been previously utilized to improve frequency resolution. Ultrasound signals have been tested in such methods resulting in higher precision frequency detection compared to common non-parametric spectral estimation methods based on the Fourier transform. Such a technique using a reversible jump Markov Chain Monte Carlo algorithm has been developed to fully characterize signals and in addition to frequency, to provide amplitude and noise estimation. The analysis of this method is demonstrated with a real copper sphere ultrasound scatter signal. Based on typical diagnostic ultrasound data between 1.2–4.5 MHz the new spectral estimation achieves 110 kHz minimum frequency resolution. This is at least twice the resolution of Fourier based methods, resulting in revealing new frequencies. The method may be used in the entire range of ultrasound imaging modalities and may help provide improved sensitivity, reproducibility and spatial resolution.

Situational awareness is achieved naturally by the human senses of sight and hearing in combination. Automatic scene understanding aims at replicating this human ability using microphones and cameras in cooperation. In this paper, audio and video signals are fused and integrated at different levels of semantic abstractions. We detect and track a speaker who is relatively unconstrained, i.e., free to move indoors within an area larger than the comparable reported work, which is usually limited to round table meetings. The system is relatively simple: consisting of just 4 microphone pairs and a single camera. Results show that the overall multimodal tracker is more reliable than single modality systems, tolerating large occlusions and cross-talk. System evaluation is performed on both single and multi-modality tracking. The performance improvement given by the audio–video integration and fusion is quantified in terms of tracking precision and accuracy as well as speaker diarisation error rate and precision–recall (recognition). Improvements vs. the closest works are evaluated: 56% sound source localisation computational cost over an audio only system, 8% speaker diarisation error rate over an audio only speaker recognition unit and 36% on the precision–recall metric over an audio–video dominant speaker recognition method.

Considering that multiple talkers may appear simultaneously, a time-frequency (TF) masking based random finite set (RFS) particle filtering (PF) method is developed for multiple acoustic source detection and tracking. The time-delay of arrival (TDOA) measurements of multiple sources are extracted by using a time-frequency masking technique, by which each source's TF bins are clustered and separated in a joint gain-ratio and time-delay histogram. Since a joint detection and tracking problem is considered, both source positions and source numbers are time-varying and need to be estimated. The tracker is built within a RFS Bayesian filtering framework. Essentially, an RFS process is used to characterize the source dynamics that include source appearance/dissappearance and motion trajectories. Latent variables are also introduced to indicate source dynamics and measurement-source associations. Subsequently, a Rao-Blackwellization PF technique is employed so that the source position state can be marginalized and only the latent variables are estimated by using the PF. The main advantage of the proposed approach is that hypothesis-pruning is formulated in a full probabilistic sense. The performance of the proposed approach is demonstrated in real speech recordings as well as in simulated room environments.

In this work we propose a novel method to automatically detect and localise the dominant speaker in an enclosed scenario by means of audio and video cues. The underpinning idea is that gesturing means speaking, so observing motions means observing an audio signal. To the best of our knowledge state-of-the-art algorithms are focussed on stationary motion scenarios and close-up scenes where only one audio source exists, whereas we enlarge the extent of the method to larger field of views and cluttered scenarios including multiple non-stationary moving speakers. In such contexts, moving objects which are not correlated to the dominant audio may exist and their motion may incorrectly drive the audio-video (AV) correlation estimation. This suggests extra localisation data may be fused at decision level to avoid detecting false positives. In this work, we learn Mel-frequency cepstral coefficients (MFCC) coefficients and correlate them to the optical flow. We also exploit the audio and video signals to estimate the position of the actual speaker, narrowing down the visual space of search, hence reducing the probability of incurring in a wrong voice-to-pixel region association. We compare our work with a state-of-the-art existing algorithm and show on real datasets a 36% precision improvement in localising a moving dominant speaker through occlusions and speech interferences.

In this paper we present a new solution to the problem of speaker tracking among people where occlusions occur (disappearance and non-speaking). In a normal conversation between two or more people, we learn speaker mel-cepstral coefficients (MFCC) and incorporate this information into a sequential Bayesian audio-video position tracker. The joint video-to-audio data association step is thus improved and we achieve robust person recognition which in turn aids tracking performance. We provide comprehensive evaluation via simulations and real data quoting tracking accuracy, precision and diarisation error rate (DER) compared to ground truth. For simulate and real experiments in an open space the trajectory tracking performance increases by 20% measured against ground truth using our approach. As a further enhancement versus the state-of-the-art, speaker identity recognition at a distance is improved by 20% by exploiting audio-video localisation cues.

In this paper we present a novel audio-visual speaker detection and localisation algorithm. Audio source position estimates are computed by a novel stochastic region contraction (SRC) audio search algorithm for accurate speaker localisation. This audio search algorithm is aided by available video information (stochastic region contraction with height estimation (SRC-HE)) which estimates head heights over the whole scene and gives a speed improvement of 56% over SRC. We finally combine audio and video data in a Kalman filter (KF) which fuses person-position likelihoods and tracks the speaker. Our system is composed of a single video camera and 16 microphones. We validate the approach on the problem of video occlusion i.e. two people having a conversation have to be detected and localised at a distance (as in surveillance scenarios vs. enclosed meeting rooms). We show video occlusion can be resolved and speakers can be correctly detected/localised in real data. Moreover, SRC-HE based joint audio-video (AV) speaker tracking outperforms the one based on the original SRC by 16% and 4% in terms of multi object tracking precision (MOTP) and multi object tracking accuracy (MOTA). Speaker change detection improves by 11% over SRC.

Reverberation and noise cause significant deterioration of audio quality and intelligibility to signals recorded in acoustic environments. Noise is usually modeled as a common signal observed in the room and independent of room acoustics. However, this simplistic model cannot necessarily capture the effects of separate noise sources at different locations in the room. This paper proposes a noise model that considers distinct noise sources whose individual acoustic impulse responses are separated into source-sensor specific and common acoustical resonances. Further to noise, the signal is distorted by reverberation. Using parametric models of the system, recursive expressions of the filtering distribution can be derived. Based on these results, a sequential Monte Carlo approach for online dereverberation and enhancement is proposed. Simulation results for speech are presented to verify the effectiveness of the model and method.

The understanding and exploitation of non-linear microbubble signals is an active research area that aims to advance contrast ultrasound into a high sensitivity and specificity diagnostic imaging modality. To discriminate the difference between echoes from tissue and contrast microbubbles, it is of significance to extract as much information of the reflected signals as possible, especially the pulse locations in the time domain and their corresponding spectral contents in the frequency domain. In this paper, a novel estimation system for extracting the information of interest is proposed. This estimation technique is based on non-parametric methods for coarse estimation, followed by a parametric method within Bayesian framework for estimation refinement. The results show that the pulse location and frequency content can be accurately estimated simultaneously. This assists in the design of transmit pulsing regimes in future work.

Acoustic reverberation introduces multipath components into an audio signal, and therefore changes the source signal statistical properties. This causes problems for source localisation and tracking since reverberation generates spurious peaks in the time delay functions, and makes the subsequent location estimator hard to track the motion trajectory. Previous time delay based tracking methods, such as the extended Kalman filter and the particle filter, are sensitive to reverberation and are unable to follow sharp changes in the source positions. In this paper, the extended Kalman filter and the particle filter are combined to solve this problem. One of the advantages of this approach is that the optimal importance function can be obtained after extended Kalman filtering. Thus, the position samples are distributed in a more accurate area than using a prior importance function. Experiment results show that the proposed algorithm outperforms the sequential importance resampling particle filter by reducing the estimation error and following the switch of speakers quickly under a moderate reverberant environment (reverberation time T60≪ 0.3s).

We propose a new image and blur prior model, based on non-stationary autoregressive (AR) models, and use these to blindly deconvolve blurred photographic images, using the Gibbs sampler. As far as we are aware, this is the first attempt to tackle a real-world blind image deconvolution (BID) problem using Markov chain Monte Carlo (MCMC) methods. We give examples with simulated and real out-of-focus images, which show the state-of-the-art results that the proposed approach provides.

Reverberation and noise cause significant deterioration of audio quality and intelligibility to signals recorded in acoustic environments. Bayesian dereverberation infers knowledge about the system by exploiting the statistical properties of speech and the acoustic channel. In Bayesian frameworks, the signal can be processed either sequentially using online methods or in a batch using offline methods. This paper compares the two approaches for blind speech dereverberation by means of a previously proposed batch approach and a novel sequential approach. Results show that while both methods have different advantages, online processing leads to a more flexible solution.

The variational Bayesian approach has recently been proposed to tackle the blind image restoration (BIR) problem. We consider extending the procedures to include realistic boundary modelling and non-stationary image restoration. Correctly modelling the boundaries is essential for achieving accurate blind restorations of photographic images, whilst nonstationary models allow for better adaptation to local image features, and therefore improvements in quality.

We present a novel method for blind image restoration which is a multidimensional extension of an approach used successfully for audio restoration. A nonstationary image model is used to increase reliability of blur estimates. This source model consists of a separate autoregressive model in each region of the image. A hierarchical Bayesian model for the observations is used, and a maximum marginalised a posteriori (MMAP) blur estimate is obtained by optimising the resulting probability density function

The all-pole model is often used to approximate rational transfer functions parsimoniously. In many applications, such as single channel blind deconvolution, an estimate of the channel is required. However, in general, attempting to model the entire channel spectrum by a single all-pole model leads to a large computational load. Hence, it is better to model a particular frequency band of the spectrum by an all-pole model, reducing a single high-dimensional optimisation to a number of low-dimensional ones. If each subband is completely decoupled from the others, and does not enforce any continuity, there are discontinuities in the spectrum at the subband boundaries. Continuity is ensured by constraining the subband parameters such that the end points at one subband boundary are matched to the spectrum in the adjacent subbands. This is formulated in the Bayesian probabilistic framework.

Blind deconvolution is fundamental in signal processing applications and, in particular, the single channel case remains a challenging and formidable problem. This paper considers single channel blind deconvolution in the case where the degraded observed signal may be modeled as the convolution of a nonstationary source signal with a stationary distortion operator. The important feature that the source is nonstationary while the channel is stationary facilitates the unambiguous identification of either the source or channel, and deconvolution is possible, whereas if the source and channel are both stationary, identification is ambiguous. The parameters for the channel are estimated by modeling the source as a time-varyng AR process and the distortion by an all-pole filter, and using the Bayesian framework for parameter estimation. This estimate can then be used to deconvolve the observed signal. In contrast to the classical histogram approach for estimating the channel poles, where the technique merely relies on the fact that the channel is actually stationary rather than modeling it as so, the proposed Bayesian method does take account for the channel's stationarity in the model and, consequently, is more robust. The properties of this model are investigated, and the advantage of utilizing the nonstationarity of a system rather than considering it as a curse is discussed.

Separability of signal mixtures given only one mixture observation is defined as the identification of the accuracy to which the signals can be separated. The paper shows that when signals are separated using the generalised Wiener filter, the degree of separability can be deduced from the filter structure. To identify this structure, the processes are represented on an arbitrary spectral domain, and a sufficient solution to the Wiener filter is obtained. The filter is composed of a term independent of the signal values, corresponding to regions in the spectral domain where the desired signal components are not distorted by interfering noise components, and a term dependent on the signal correlations, corresponding to the region where components overlap. An example of determining perfect separability of modulated random signals is given.