We introduce ImmerseDiffusion, an end-to-end generative audio model that produces 3D immersive soundscapes conditioned on the spatial, temporal, and environmental conditions of sound objects. ImmerseDiffusion is trained to generate first-order ambisonics (FOA) audio, which is a conventional spatial audio format comprising four channels that can be rendered to multichannel spatial output. The proposed generative system is composed of a spatial audio codec that maps FOA audio to latent components, a latent diffusion model trained based on various user input types, namely, text prompts, spatial, temporal and environmental acoustic parameters, and optionally a spatial audio and text encoder trained in a Contrastive Language and Audio Pretraining (CLAP) style. We propose metrics to evaluate the quality and spatial adherence of the generated spatial audio. Finally, we assess the model performance in terms of generation quality and spatial conformance, comparing the two proposed modes: "descriptive", which uses spatial text prompts) and "parametric", which uses non-spatial text prompts and spatial parameters. Our evaluations demonstrate promising results that are consistent with the user conditions and reflect reliable spatial fidelity.

We study the problem of stereo singing voice cancellation, a subtask of music source separation, whose goal is to estimate an instrumental background from a stereo mix. We explore how to achieve performance similar to large state-of-the-art source separation networks starting from a small, efficient model for real-time speech separation. Such a model is useful when memory and compute are limited and singing voice processing has to run with limited look-ahead. In practice, this is realised by adapting an existing mono model to handle stereo input. Improvements in quality are obtained by tuning model parameters and expanding the training set. Moreover, we highlight the benefits a stereo model brings by introducing a new metric which detects attenuation inconsistencies between channels. Our approach is evaluated using objective offline metrics and a large-scale MUSHRA trial, confirming the effectiveness of our techniques in stringent listening tests.

We propose dynamic curriculum learning via data parameters for noise robust keyword spotting. Data parameter learning has recently been introduced for image processing, where weight parameters, so-called data parameters, for target classes and instances are introduced and optimized along with model parameters. The data parameters scale logits and control importance over classes and instances during training, which enables automatic curriculum learning without additional annotations for training data. Similarly, in this paper, we propose using this curriculum learning approach for acoustic modeling, and train an acoustic model on clean and noisy utterances with the data parameters. The proposed approach automatically learns the difficulty of the classes and instances, e.g. due to low speech to noise ratio (SNR), in the gradient descent optimization and performs curriculum learning. This curriculum learning leads to overall improvement of the accuracy of the acoustic model. We evaluate the effectiveness of the proposed approach on a keyword spotting task. Experimental results show 7.7% relative reduction in false reject ratio with the data parameters compared to a base-line model which is simply trained on the multiconditioned dataset.

In distributed microphone arrays (DMAs) the source location information can be defined at the intra and inter-node levels. Indeed, while the first type of information results from the diversity of acoustic channels recorded by microphones embedded in the same node, the second is attributed to the differences between the acoustic channels observed by spatially distributed nodes. Both cues are very useful in DMA processing, and the aim of this paper is to utilize both of them to cluster and separate multiple competing speech signals. To capture the intra-node information, we employ the normalized recording vector, while at the inter-node level, we consider different features including the energy level differences with and without the phase differences between nodes. We model the intra-node information using the Watson mixture model (WMM), and propose using the Gamma mixture model (GaMM), Dirichlet mixture model (DMM), and WMM to model different inter-node location features. Furthermore, we propose several integrations of the intra-node and inter-node feature contributions to cluster speech recordings using the expectation maximization algorithm. Finally, simulation results are provided to demonstrate the performance of all ensuing methods.

This paper discusses microphone array based interference reduction approaches for robust automatic speech recognition. A model based multichannel spectral enhancement approach has recently been proposed for effectively reducing interference by exploiting both the spatial and spectral features of the signals. With the goal of further improving the effectiveness of this approach, we propose a new framework that combines this approach with a microphone-array based beamforming approach. Because the two approaches can work in a complementary manner in the proposed framework, they can greatly improve the interference reduction performance. We apply the proposed framework to the recognition of actual meetings, and show that it is superior to the use of beamforming or spectral enhancement alone in terms of the word error rates.

We propose a new approach for clustering competing speech sources using distributed microphone arrays. In this approach, we first define two feature vectors where the first captures the intra-node location information while the second captures the level difference of speech energy recorded at different nodes. Then, we introduce Watson and Dirichlet mixture models to model the first and second features, respectively. We integrate both types of information in an expectation maximization algorithm to cluster the simultaneous speech sources. The performance of the proposed approach is superior to best node selection and comparable to centralized processing in terms of conventional blind source separation metrics.

We propose a new framework for joint multichannel speech source separation and acoustic noise reduction. In this framework, we start by formulating the minimum-mean-square error (MMSE)-based solution in the context of multiple simultaneous speakers and background noise, and outline the importance of the estimation of the activities of the speakers. The latter is accurately achieved by introducing a latent variable that takes N+1 possible discrete states for a mixture of N speech signals plus additive noise. Each state characterizes the dominance of one of the N+1 signals. We determine the posterior probability of this latent variable, and show how it plays a twofold role in the MMSE-based speech enhancement. First, it allows the extraction of the second order statistics of the noise and each of the speech signals from the noisy data. These statistics are needed to formulate the multichannel Wiener-based filters (including the minimum variance distortionless response). Second, it weighs the outputs of these linear filters to shape the spectral contents of the signals' estimates following the associated target speakers' activities. We use the spatial and spectral cues contained in the multichannel recordings of the sound mixtures to compute the posterior probability of this latent variable. The spatial cue is acquired by using the normalized observation vector whose distribution is well approximated by a Gaussian-mixture-like model, while the spectral cue can be captured by using a pre-trained Gaussian mixture model for the log-spectra of speech. The parameters of the investigated models and the speakers' activities (posterior probabilities of the different states of the latent variable) are estimated via expectation maximization. Experimental results including comparisons with the well-known independent component analysis and masking are provided to demonstrate the efficiency of the proposed framework.

In this paper, we propose a new framework to separate multiple speech signals and reduce the additive acoustic noise using multiple microphones. In this framework, we start by formulating the minimum-mean-square error (MMSE) criterion to retrieve each of the desired speech signals from the observed mixtures of sounds and outline the importance of multi-speaker activity detection. The latter is modeled by introducing a latent variable whose posterior probability is computed via expectation maximization (EM) combining both the spatial and spectral cues of the multichannel speech observations. We experimentally demonstrate that the resulting joint blind source separation (BSS) and noise reduction solution performs remarkably well in reverberant and noisy environments.

We propose a new approach for online noise power spectral density (psd) tracking. In this approach, the prior and posterior probabilities of speech absence and also noise statistics are analytically retrieved from a maximum-likelihood-based criterion at every time-frequency slot. The recursive update rules of these three terms are performed in a unified manner and without relying on the conventional tracking of speech psd minima. A single parameter (a forgetting factor) is needed in this process. Comparisons with state of the art methods demonstrate the effectiveness of our proposal.

In this correspondence, we study the performance of differential microphone arrays (DMAs) in terms of noise reduction, speech distortion, and signal-to-noise ratio (SNR) gain. We also investigate their beampatterns and array gains. We start by establishing the expressions of these performance measures involving general derivatives of the channel transfer functions. Afterwards, we specify our results in the case of anechoic near-field and far-field propagation models.

Noise statistics estimation is a paramount issue in the design of reliable noise-reduction algorithms. Although significant efforts have been devoted to this problem in the literature, most developed methods so far have focused on the single-channel case. When multiple microphones are used, it is important that the data from all the sensors are optimally combined to achieve judicious updates of the noise statistics and the noise-reduction filter. This contribution is devoted to the development of a practical approach to multichannel noise tracking and reduction. We combine the multichannel speech presence probability (MC-SPP) that we proposed in an earlier contribution with an alternative formulation of the minima-controlled recursive averaging (MCRA) technique that we generalize from the single-channel to the multichannel case. To demonstrate the effectiveness of the proposed MC-SPP and multichannel noise estimator, we integrate them into three variants of the multichannel noise reduction Wiener filter. Experimental results show the advantages of the proposed solution.

The localization process consists in finding the candidate source location that maximizes the synchrony between the properly time-shifted microphone outputs. In addition to using well-known crosscorrelation-based criteria such as the steered response power (SRP), minimum variance (MV), and multichannel crosscorrelation (MCCC), this synchrony can be measured using the averaged magnitude difference function (AMDF) and the averaged magnitude sum function (AMSF) whose calculations involve low computational cost. In this paper, we study the crosscorrelation and AMDF (with AMSF) based approaches using an arbitrary number of microphones. Specifically, we use the eigenanalysis of the parameterized spatial correlation matrix (PSCM) to first provide a unifying study of the most popular crosscorrelation-based techniques. Then, we show the efficiency of the AMDF and AMSF in localizing an acoustic source using multiple microphones by proposing two new parameterized matrices named as the parameterized averaged magnitude difference matrix (PAMDM) and the parameterized averaged magnitude sum matrix (PAMSM). The eigenanalysis of these two matrices reveals new criteria.

We study the linearly constrained minimum variance (LCMV) and the minimum variance distortionless response (MVDR) filters when multiple interferers and unknown (ambient) noise coexist with a target speech signal. Precisely, the LCMV is designed to remove all the interference signals while preserving the desired speech and attempting to reduce the ambient noise components. The MVDR is simply formulated such that the overall ambient-noise-plus-interference are reduced while satisfying a distortionless constraint. We provide simplified expressions for both beamformers and show their relationship. Furthermore, we underline the limitations of the LCMV when the ambient noise is present. When the latter is absent, we also prove that the MVDR degenerates to the LCMV. Numerical examples are provided to support our study.

The parameterized multichannel Wiener filter (PMWF) is known to allow for a flexible tuning of speech distortion and noise reduction. In addition, the output signal-to-noise ratio (SNR) is a natural metric that shows the true effect of such a filter on both residual noise and filtered speech. Earlier contributions have only shown that the output SNR of the Wiener filter is larger than the input SNR (as a proof of its effectiveness). However, the effect of the tuning parameter (denoted as ß following the notation of [1]) on this metric is not yet understood. In this paper, we prove that the global (fullband) output SNR of the PMWF is an increasing function of ß but remains below an asymptotic value. As a byproduct, a very simplified proof of the global output SNR improvement is provided.

Several contributions have been made so far to develop optimalmultichannellinear filtering approaches and show their ability to reduce the acoustic noise. However, there has not been a clearunifying theoreticalanalysis of their performance in terms of both noise reduction and speech distortion. To fill this gap, we analyze the frequency-domain (non-causal) multichannel linear filtering for noise reduction in this paper. For completeness, we consider the noise reduction constrained optimization problem that leads to the parameterized multichannel non-causal Wiener filter (PMWF). Our contribution is fivefold. First, we formally show that the minimum variance distortionless response (MVDR) filter is a particular case of the PMWF by properly formulating the constrained optimization problem of noise reduction. Second, we propose new simplified expressions for the PMWF, the MVDR, and the generalized sidelobe canceller (GSC) that depend on the signals' statistics only. In contrast to earlier works, these expressions are explicitly independent of the channel transfer function ratios. Third, we quantify the theoretical gains and losses in terms of speech distortion and noise reduction when using the PWMF by establishing new simplified closed-form expressions for three performance measures, namely, the signal distortion index, the noise reduction factor (originally proposed in the paper titled ldquoNew insights into the noise reduction Wiener filter,rdquo by J. Chen (IEEE Transactions on Audio, Speech, and Language Processing, Vol. 15, no. 4, pp. 1218-1234, Jul. 2006) to analyze the single channel time-domain Wiener filter), and the output signal-to-noise ratio (SNR). Fourth, we analyze the effects of coherent and incoherent noise in addition to the benefits of utilizing multiple microphones. Fifth, we propose a new proof for thea posterioriSNR improvement achieved by the PMWF. Finally, we provide some simulations results to corroborate the findings of this work.

Broadband source localization has several applications ranging from automatic video camera steering to target signal tracking and enhancement through beamforming. Consequently, there has been a considerable amount of effort to develop reliable methods for accurate localization over the last few decades. Essentially, the localization process consists in finding the candidate source location that maximizes the synchrony between the properly time-shifted microphone outputs. In addition to using well known cross-correlation-based criteria such as the steered response power (SRP), minimum variance (MV), and multichannel cross-correlation (MCCC), this synchrony can also be measured using the averaged magnitude difference function (AMDF) and the averaged magnitude sum function (AMSF) whose calculations involve low computational cost. In earlier related works, the latter techniques have been used for time delay estimation (TDE) of a target source observed by only one pair of microphones. Their generalization to the multiple microphone case and application to source localization have not been studied yet. In this paper, we consider both categories, i.e., cross-correlation and AMDF (with AMSF)-based approaches, using an arbitrary number of microphones, and analyze their performance. Specifically, we first provide a unifying study of the most popular cross-correlation-based techniques, such as the SRP, MV, and MCCC. In this paper, we use the eigenanalysis of the parameterized spatial correlation matrix (PSCM) to classify these methods and gain some insight into their performance. We demonstrate, for instance, that the MV and SRP consist in searching the major eigenvalue of the PSCM, while the MCCC, essentially, combines its minor eigenvalues when scanning for the source location. Inspired by this analysis, we show, in the second part of this work, the efficiency of the AMDF and AMSF in localizing an acoustic source using multiple microphones. Indeed, we propose two new parameterized matrices named as the parameterized averaged magnitude difference matrix (PAMDM) and the parameterized averaged magnitude sum matrix (PAMSM). The eigenanalysis of these matrices also reveals new criteria for acoustic source localization. Simulation results are provided to illustrate the effectiveness of all the investigated and proposed methods.

The knowledge of the target speech presence probability in a mixture of signals captured by a speech communication system is of paramount importance in several applications including reliable noise reduction algorithms. In this correspondence, we establish a new expression for speech presence probability when an array of microphones with an arbitrary geometry is used. Our study is based on the assumption of the Gaussian statistical model for all signals and involves the noise and noisy data statistics only. In comparison with the single-channel case, the new proposed multichannel approach can significantly increase the detection accuracy. In particular, when the additive noise is spatially coherent, perfect speech presence detection is theoretically possible, while when the noise is spatially white, a coherent summation of speech components is performed to allow for enhanced speech presence probability estimation.

We investigate a general framework for noise reduction which consists in controlling the level of signal distortion while reducing the level of noise. A parameterized non-causal filter that allows for tuning the signal distortion and noise reduction inversely is obtained and is referred to as parameterized multichannel non-causal Wiener filter (PMWF) herein. The same optimization problem leads to the minimum variance distortionless response (MVDR) as a particular case of the PMWF. In contrast to earlier works, the proposed expressions of the PMWF and MVDR are simplified and require the knowledge of the speech and noise statistics only. To rigorously quantify the gains and losses when using these filters, we establish simplified closed-form expressions for three measures, namely, the signal distortion index, the noise reduction factor, and the output signal-to-noise ratio (SNR), and highlight the tradeoff between noise reduction and speech distortion in the multichannel case.

Acoustic echoes represent a major source of discomfort in hands free, full-duplex, communication systems. The problem becomes particularly difficult when the loudspeakers are nonlinear as considered in this paper. In contrast to the single-microphone linear and nonlinear acoustic echo cancellation techniques, we take advantage of the spatial diversity offered by the microphone arrays. Indeed, having a set of microphones and multiple sources (i.e., the near and far ends) that can be active at the same time, this problem can be solved using a blind source separation (BSS) algorithm. The performance of the BSS can be further improved when combined with a linear acoustic echo canceler (LAEC). In this paper, we study the potentials of joint BSS and LAEC to cancel the echo signals in two schemes. In the first scheme, the BSS is deployed as a front-end and has a twofold function: reducing the acoustic echo and creating a linearly transformed echo reference that is used by the LAEC as a post-processor. In the second scheme, the BSS operates on multiple LAECs outputs to further reduce the residual echo from the target signal. We show that the first scheme outperforms the second one.

We propose a new method for noise reduction using a microphone array. The method takes advantage of the spatial diversity inherent to microphone arrays and optimizes certain criteria, namely, the output signal to noise ratio (SNR) or the mean squared error (MSE), subject to the constraint of spatial prediction that relates the noise free signals captured by the microphones. Simulation results demonstrate that resorting to this new method leads to high rate of noise reduction and low signal distortion.

An exhaustive investigation of the overdetermined blind source separation (BSS) 3 times 2 problem is presented. We establish a new relationship between the column vectors of the channel matrix using second order statistics (SOS) only. This relationship is then exploited to factorize the separating matrix into a whitening term, whose expression depends on whether the observations are corrupted by noise or not, and a 2 times 2 orthogonal matrix that can be determined using high order statistics (HOS). Simulation results demonstrate that resorting to the new form of the whitening matrix, mainly in the case of noisy ill-conditioned BSS, results in improved performance compared to the common whitening-based techniques.