Dialog Enhancement (DE) is a feature which allows a user to increase the level of dialog in TV or movie content relative to nondialog sounds. When only the original mix is available, DE is "unguided," and requires source separation. In this paper, we describe the DeepSpace system, which performs source separation using both dynamic spatial cues and source cues to support unguided DE. Its technologies include spatio-level filtering (SLF) and deep-learning based dialog classification and denoising. Using subjective listening tests, we show that DeepSpace demonstrates significantly improved overall performance relative to state-of-the-art systems available for testing. We explore the feasibility of using existing automated metrics to evaluate unguided DE systems.

Postfilters are commonly used in speech coding for the attenuation of quantization noise. In the presence of acoustic background noise or distortion due to tandeming operations, the postfilter parameters are not adjusted and the performance is, therefore, not optimal. We propose a modification that consists of replacing the nonadaptive postfilter parameters with parameters that adapt to variations in spectral flatness, obtained from the noisy speech. This generalization of the postfiltering concept can handle a larger range of noise conditions, but has the same computational complexity and memory requirements as the conventional postfilter. Test results indicate that the presented algorithm improves on the standard postfilter, as well as on the combination of a noise attenuation preprocessor and the conventional postfilter.

A flexible and low-complexity entropy-constrained vector quantizer (ECVQ) scheme based on Gaussian mixture models (GMMs), lattice quantization, and arithmetic coding is presented. The source is assumed to have a probability density function of a GMM. An input vector is first classified to one of the mixture components, and the Karhunen-Loeve transform of the selected mixture component is applied to the vector, followed by quantization using a lattice structured codebook. Finally, the scalar elements of the quantized vector are entropy coded sequentially using a specially designed arithmetic coder. The computational complexity of the proposed scheme is low, and independent of the coding rate in both the encoder and the decoder. Therefore, the proposed scheme serves as a lower complexity alternative to the GMM based ECVQ proposed by Gardner, Subramaniam and Rao. The performance of the proposed scheme is analyzed under a high-rate assumption, and quantified for a given GMM. The practical performance of the scheme was evaluated through simulations on both synthetic and speech line spectral frequency (LSF) vectors. For LSF quantization, the proposed scheme has a comparable performance to at rates relevant for speech coding (20-28 bits per vector) with lower computational complexity.

In this paper, we present a scalable entropy-constrained vector quantizer based on Gaussian mixture models (GMMs), lattice quantization, and arithmetic coding. We assume that the source has a probability density function of a GMM. The scheme is based on a mixture component classifier, the Karhunen Loeve transform of the component, followed by a lattice quantization. The scalar elements of the quantized vector are entropy coded using a specially designed arithmetic coder. The proposed scheme has a computational complexity that is independent of rate, and quadratic with respect to vector dimension. The design is flexible and allows for adjusting the desired target rate on-the-fly. We evaluated the performance of the proposed scheme on speech-derived source vectors. It was demonstrated that the proposed scheme outperforms a fixed-rate GMM based vector quantizer, and performs closely to the theoretical optimum.

In this paper, we propose a Bayesian minimum mean squared error approach for the joint estimation of the short-term predictor parameters of speech and noise, from the noisy observation. We use trained codebooks of speech and noise linear predictive coefficients to model the a priori information required by the Bayesian scheme. In contrast to current Bayesian estimation approaches that consider the excitation variances as part of the a priori information, in the proposed method they are computed online for each short-time segment, based on the observation at hand. Consequently, the method performs well in nonstationary noise conditions. The resulting estimates of the speech and noise spectra can be used in a Wiener filter or any state-of-the-art speech enhancement system. We develop both memoryless (using information from the current frame alone) and memory-based (using information from the current and previous frames) estimators. Estimation of functions of the short-term predictor parameters is also addressed, in particular one that leads to the minimum mean squared error estimate of the clean speech signal. Experiments indicate that the scheme proposed in this paper performs significantly better than competing methods

Kalman filtering is a powerful technique for the estimation of a signal observed in noise that can be used to enhance speech observed in the presence of acoustic background noise. In a speech communication system, the speech signal is typically buffered for a period of 10-40 ms and, therefore, the use of either a causal or a noncausal filter is possible. We show that the causal Kalman algorithm is in conflict with the basic properties of human perception and address the problem of improving its perceptual quality. We discuss two approaches to improve perceptual performance. The first is based on a new method that combines the causal Kalman algorithm with pre- and postfiltering to introduce perceptual shaping of the residual noise. The second is based on the conventional Kalman smoother. We show that a short lag removes the conflict resulting from the causality constraint and we quantify the minimum lag required for this purpose. The results of our objective and subjective evaluations confirm that both approaches significantly outperform the conventional causal implementation. Of the two approaches, the Kalman smoother performs better if the signal statistics are precisely known, if this is not the case the perceptually weighted Kalman filter performs better.

In this paper, we present a new technique for the estimation of short-term linear predictive parameters of speech and noise from noisy data and their subsequent use in waveform enhancement schemes. The method exploits a priori information about speech and noise spectral shapes stored in trained codebooks, parameterized as linear predictive coefficients. The method also uses information about noise statistics estimated from the noisy observation. Maximum-likelihood estimates of the speech and noise short-term predictor parameters are obtained by searching for the combination of codebook entries that optimizes the likelihood. The estimation involves the computation of the excitation variances of the speech and noise auto-regressive models on a frame-by-frame basis, using the a priori information and the noisy observation. The high computational complexity resulting from a full search of the joint speech and noise codebooks is avoided through an iterative optimization procedure. We introduce a classified noise codebook scheme that uses different noise codebooks for different noise types. Experimental results show that the use of a priori information and the calculation of the instantaneous speech and noise excitation variances on a frame-by-frame basis result in good performance in both stationary and nonstationary noise conditions.

The Kalman recursion is a powerful technique for reconstruction of the speech signal observed in additive background noise. In contrast to Wiener filtering and spectral subtraction schemes, the Kalman algorithm can be easily implemented in both causal and noncausal form. After studying the perceptual differences between these two implementations we propose a novel algorithm that combines the low complexity and the robustness of the Kalman filter and the proper noise shaping of the Kalman smoother.

In this paper, we propose a Bayesian approach for the estimation of the short-term predictor parameters of speech and noise, from the noisy observation. The resulting estimates of the speech and noise spectra can be used in a Wiener filter or any state-of-the-art speech enhancement system. We utilize a-priori information about both speech and noise in the form of trained codebooks of linear predictive coefficients. In contrast to current Bayesian estimation approaches that consider the excitation variances as part of the a-priori information, in the proposed method they are computed analytically based on the observation at hand. Consequently, the method performs well in nonstationary noise conditions. Experimental results confirm the superior performance of the proposed method compared to existing Bayesian approaches, such as those based on hidden Markov models.

In this paper we address the problem of vector quantization of speech in a noisy environment. We show that the performance of a vector quantization system can be improved by adapting the distortion measure to the changing environmental conditions. The proposed method emphasizes the distortion in spectral regions where the speech signal dominates. The method functions well even when conventional pre-processor methods fail because the noise statistics cannot be estimated reliably from speech pauses (as, e.g., in tandeming operations). Objective tests confirm that the use of environmentally adaptive measures significantly improves estimation accuracy in noisy speech, while preserving the quality in the case of clean input. 1.

An algorithm for multiple description coding (MDC) based on Gaussian mixture models (GMMs) is presented. Based on the parameters of the GMM, the algorithm combines MDC scalar quantizers, yielding a source-optimized vector MDC system. The performance is evaluated on a speech spectrum source in terms of mean-squared error and log spectral distortion. It is demonstrated experimentally that the proposed system outperforms single description coding and repetition coding over a wide range of channel failure probabilities. The proposed algorithm has a complexity that is linear in rate and dimension while retaining a near optimal vector quantizer point density.

The paper introduces a modification of the commonly used postfilter that improves performance when acoustic background noise is present. The modification consists of replacing the nonadaptive postfilter parameters that govern the degree of spectral emphasis (commonly denoted as /spl gamma//sub 1/ and /spl gamma//sub 2/) with parameters that adapt to the noise statistics. We describe an effective mapping from the noise statistics to the emphasis parameters and provide a low complexity noise estimation algorithm that is sufficient for this application. The resulting noise-adaptive postfilter successfully attenuates the background noise and naturally converges to the conventional postfilter at high SNR conditions. Thus, the speech enhancement problem is solved with minimal modification of legacy codecs, since the existing structure of the speech codec is used. Test results indicate that the presented algorithm significantly outperforms the standard postfilter with non-adaptive parameters.

Waveform quantization of speech using Gaussian mixture models (GMM) is proposed. GMM are trained directly on the speech waveform, and high dimensional vector quantizers (VQ) that efficiently exploit the redundancy are constructed based on the GMM parameters. Two types of GMM are studied. The complexity of the scheme is independent of the rate, and the rate can be changed without retraining the VQ. A shape-gain structure improves performance and robustness. Pre- and post-processing using spectral amplitude warping further improves perceptual quality. A 32-dimensional VQ operating at 2 bits/sample reproduces speech sampled at 8 kHz with a PESQ score of 4.2.

We describe a technique for obtaining estimates of the short-term predictor parameters of speech under noisy conditions. We use a-priori information about speech in the form of a trained codebook of speech linear predictive coefficients. Our contribution is two-fold. First, we provide a framework where the standard vector quantization search to obtain the quantized linear predictive coefficients can be replaced by a maximum likelihood search, given the noisy observation, the speech codebook and an estimate of the noise. This results in an enhancement method that is integrated with parametric coders such as linear predictive analysis-by-synthesis coders. Second, we provide a scheme where the chosen vector is not restricted to be an element of the codebook. An interpolative search between the maximum likelihood estimate and its nearest neighbors in the codebook is used to improve the precision of the estimated parameters. Such a scheme is relevant when enhancement is considered separately from coding. Experimental results show improved performance for the proposed methods.

Lately, Gaussian mixture (GM) models have found new applications in speech processing, and particularly in speech coding. This paper provides a review of GM based quantization and prediction. The main contribution is a discussion on GM model optimization. Two previously presented algorithms of EM-type are analyzed in some detail, and models are estimated and evaluated experimentally using theoretical measures as well as GM based speech spectrum coding and prediction. It has been argued that since many sources have a bounded support, this should be utilized in both the choice of model, and the optimization algorithm. By low-dimensional modeling examples, illustrating the behavior of the two algorithms graphically, and by full-scale evaluation of GM based systems, the advantages of a bounded support approach are quantified. For all evaluation techniques in the study, model accuracy is improved when the bounded support approach is adopted. The gains are typically largest for models with diagonal covariance matrices.

Multidimensional companding quantization is analyzed theoretically for the case of high-resolution and mean-squared-error distortion. The optimality of choosing the expander function to be the inverse of the compressor function is established first. Heuristic derivations of the point density and moment of inertia of companding are given, which combined result in the distortion formula by Bucklew. Further, the interaction between the lattice quantizer (LQ) and compressor function is studied. Optimality is achieved with a second moment optimal LQ, shaped by a compressor-dependent linear transform. High rate theory for a radial compander and spherically symmetric sources is reviewed. The result is used to evaluate the performance on an independent and identically distributed (i.i.d.) Gaussian source. The radial compander clearly outperforms both a product scalar quantizer and a spherical quantizer for dimensions higher than two. Radial companding is also generalized to the correlated Gaussian source. Finally, a comparison of theory and practical companders is made in a Gaussian framework.

A theoretical analysis of recursive speech spectrum coding, where predictive and finite state schemes are special cases, is presented. We evaluate the spectral distortion (SD) theoretically and design coders that minimize the SD. The analysis rests on three cornerstones: high-rate theory, PDF modeling, and an approximation of SD. A derivation of the mean L/sub 2/-norm distortion of a recursive quantizer operating at high rate is provided. Also, the distortion distribution is supplied. The evaluation of the distortion expressions requires a model of the joint PDF of two consecutive spectrum vectors. The LPC spectrum source considered here has outcomes in a bounded region, and this is taken into account in the choice of model and modeling algorithm. It is further shown how to approximate the SD with an L/sub 2/-norm measure. Combining the results, we show theoretically that 16 bits are needed to achieve an average SD of 1 dB when quantizing ten-dimensional (10-D) spectrum vectors using a first-order recursive scheme. A gain of six bits per frame is noted compared to memoryless quantization. These results rely on high-rate assumptions which are validated in experiments. There, actual high-rate optimal coders are designed and evaluated.

We propose a recursive coding scheme for spectral parameters. The coder utilizes a Gaussian mixture model describing the PDF of pairs of source vectors. From this model we can, at each coding instant, calculate a conditional PDF for the vector to be coded given the previously quantized vector. Knowing this conditional PDF, we can use high rate theory to design an optimal codebook. To reduce complexity we employ a classified vector quantizer structure. The mixture components with highest probability around the incoming vector are detected and the corresponding codebooks are searched. A pair of indices, representing the chosen component and codevector, is transmitted. Experiments show a small degradation in performance due to the imposed structural constraint. When compared with a standard predictive vector quantization (PVQ), slightly superior performance, both objectively and subjectively, is observed. Further, close to transparent spectral coding is achieved at 16 bits per frame when employed in a sinusoidal vocoder.

This paper presents a method for obtaining numerical estimates of high rate vector quantization (VQ) performance suitable for sources for which the PDF is not analytically available. In the proposed method, the VQ point density is described from a Gaussian mixture model optimized for the data. Employing this method for LPC spectrum quantization, we obtain high rate expressions for both the average spectral distortion (SD) and the distribution function of the SD. We estimate the minimum bits required for a quantizer to obtain an average SD of 1 dB and the outlier statistics for that quantizer. We find that approximately 3 bits can be saved as compared to a 2-split LSF-based vector quantizer.