Keyword spotting (KS) and automatic speech recognition (ASR) on smart speakers in a home environment with interfering signals from loudspeakers are challenging tasks to this day, despite improvements in acoustic echo cancellation (AEC) systems. In this work we propose to combine a single microphone AEC system, consisting of an adaptive linear filter (linear AEC) and a neural echo suppressor (NES), with an adaptive filter developed for multi-microphone noise reduction, called Cleaner. This additional enhancement step allows the AEC system to profit from spatial information to remove residual echo. The single microphone NES model improves upon the waveform domain counterpart proposed in [1] using a frequency domain representation that helps with generalization. Furthermore, we show that using multiple linear AEC configurations during model training provides large gains over a fixed configuration. On the hardest considered test condition, the proposed system outperforms the baseline model [1] for single microphone input by 66 % (relative) in KS false reject rate (FRR) and 52 % (relative) in ASR word error rate (WER). Using the multi-microphone setting, the FRR is reduced by an additional 52 % and the WER by an additional 32 %.

Acoustic Echo Cancellation (AEC) is essential for accurate recognition of queries spoken to a smart speaker that is playing out audio.Previous work has shown that a neural AEC model operating on log-mel spectral features (denoted "logmel" hereafter) can greatly improve Automatic Speech Recognition (ASR) accuracy when optimized with an auxiliary loss utilizing a pre-trained ASR model encoder.In this paper, we develop a conformer-based waveform-domain neural AEC model inspired by the "TasNet" architecture.The model is trained by jointly optimizing Negative Scale-Invariant SNR (SISNR) and ASR losses on a large speech dataset.On a realistic rerecorded test set, we find that cascading a linear adaptive AEC and a waveform-domain neural AEC is very effective, giving 56-59% word error rate (WER) reduction over the linear AEC alone.On this test set, the 1.6M parameter waveform-domain neural AEC also improves over a larger 6.5M parameter logmeldomain neural AEC model by 20-29% in easy to moderate conditions.By operating on smaller frames, the waveform neural model is able to perform better at smaller sizes and is better suited for applications where memory is limited.

We consider the problem of recognizing speech utterances spoken to a device which is generating a known sound waveform; for example, recognizing queries issued to a digital assistant which is generating responses to previous user inputs. Previous work has proposed building acoustic echo cancellation (AEC) models for this task that optimize speech enhancement metrics using both neural network as well as signal processing approaches.Since our goal is to recognize the input speech, we consider enhancements which improve word error rates (WERs) when the predicted speech signal is passed to an automatic speech recognition (ASR) model. First, we augment the loss function with a term that produces outputs useful to a pre-trained ASR model and show that this augmented loss function improves WER metrics. Second, we demonstrate that augmenting our training dataset of real world examples with a large synthetic dataset improves performance. Crucially, applying SpecAugment style masks to the reference channel during training aids the model in adapting from synthetic to real domains. In experimental evaluations, we find the proposed approaches improve performance, on average, by 57% over a signal processing baseline and 45% over the neural AEC model without the proposed changes.

This paper presents a novel dual-microphone speech enhancement algorithm to improve noise robustness of hotword (wake-word) detection as a special application of keyword spotting. It exploits two unique properties of hotwords: they are leading phrases of valid voice queries that we intend to respond and have short durations. Consequently an STFT-based adaptive noise cancellation method modified to use deferred filter coefficients is proposed to extract hotwords out from noisy stereo microphone signals. The new algorithm is tested with two considerably different neural hotword detectors. Both systems have significantly reduced the false-reject rate when background has strong TV noise.

This paper presents a robust, small-footprint, far-field keyword spotting (KWS) algorithm, which was inspired by the human auditory system's ability to achieve the so-called cocktail party effect in adverse acoustic environments. It introduces the idea of combining microphone-array speech enhancement with machine learning, by incorporating a feedback path from the neural network (NN) KWS classifier to its signal preprocessing frontend so that frontend noise reduction can benefit from, and in turn, better serve backend machine intelligence. We find that the new system can significantly improve KWS performance for Google Home when there is strong music or TV noise in the background. While this innovative and successfully validated strategy of combining signal processing and machine learning is developed for KWS, its technical feasibility is presumably extensible to many other applications, including noise robust speaker identification and automatic speech recognition.

This letter presents a phase-sensitive joint learning algorithm for single-channel speech enhancement. Although a deep learning framework that estimates the time-frequency (T-F) domain ideal ratio masks demonstrates a strong performance, it is limited in the sense that the enhancement process is performed only in the magnitude domain, while the phase spectra remain unchanged. Thus, recent studies have been conducted to involve phase spectra in speech enhancement systems. A phase-sensitive mask (PSM) is a T-F mask that implicitly represents phase-related information. However, since the PSM has an unbounded value, the networks are trained to target its truncated values rather than directly estimating it. To effectively train the PSM, we first approximate it to have a bounded dynamic range under the assumption that speech and noise are uncorrelated. We then propose a joint learning algorithm that trains the approximated value through its parameterized variables in order to minimize the inevitable error caused by the truncation process. Specifically, we design a network that explicitly targets three parameterized variables: 1) speech magnitude spectra; 2) noise magnitude spectra; and 3) phase difference of clean to noisy spectra. To further improve the performance, we also investigate how the dynamic range of magnitude spectra controlled by a warping function affects the final performance in joint learning algorithms. Finally, we examined how the proposed additional constraint that preserves the sum of the estimated speech and noise power spectra affects the overall system performance. The experimental results show that the proposed learning algorithm outperforms the conventional learning algorithm with the truncated phase-sensitive approximation.

In this paper, we investigate the effect of temporal correlation on the dependence between the speech narrow and high frequency bands covering the 0.3-3.4 kHz and 3.7-8 kHz ranges, respectively. We follow the technique of using Gaussian mixture modelling of spectral envelopes represented by Mel-frequency cepstral coefficients. The correlation between the disjoint speech frequency bands is quantified through mutual information (MI) and its ratio to highband entropy. Speech exhibits considerable temporal correlation that is not explicitly accounted for by static parametrization of spectral envelopes. Including memory in speech parametrization (through delta features) incorporates such temporal information of speech in its modelling, and hence, MI gains are to be expected resulting in bandwidth extension with better performance. Results show that exploiting delta features can increase certainty about the highband (ratio of MI to highband entropy) by as much as 216% relatively, corresponding to an absolute increase of 12%

A novel perceptual postfilter is introduced. For each frame, the filter gains, z, are estimated given a vector, y, of the quantized LSFs and the long-term prediction gain of the corresponding frame. The proposed perceptual postfilter is derived from an optimal MMSE estimator, i.e. the estimated gain vector is ˆz = E{z|y}. The MMSE estimator is based on the conditional pdf of z given y, which is computed from the joint pdf modelled by a GMM. The proposed perceptual postfilter improves the speech naturalness comparing with the conventional adaptive postfilter, while maintaining the property of being an “add-on ” postfilter without modification to the current encoder. 1

We present a fixed rate encoding scheme for the line spectrum pair (LSP) representation of an LPC-filter, based on Gaussian mixture (GM) modeling. For each mixture component, we construct a codebook by a union of product quantizers. Each local codebook is trained, independently, using a clustering scheme similar to the generalized Lloyd algorithm (GLA), over synthetic data. The training algorithm iterates fast, due to low complexity encoding, and converges in few iterations. The overall codebook is a combination of local codebooks, and inherits their high performance, while having a moderate complexity. We provide numerical results for average spectral distortion (SD) of the proposed encoder, and benchmark them by a lower bound, according to high-rate theory. We achieve an average SD (full-band measure) of 1 dB at 23 b/frame, for speech signals sampled at 8 kHz and LPC of order 10. By tolerating additional complexity, we reach a SD within 0.01 dB of the lower bound.

An encoding scheme is presented based on random coding theory in conjunction with companding techniques. This combination provides an easy design, and a low-storage quantizer. The groundwork of the study is Gaussian sources. The motivation is the attractive properties of Gaussian and Gaussian mixtures (GMs), and their applications in data compression. Here, codebooks are generated by companding a union of randomly translated cubic lattices. The compander functions operate scalar wise, providing an affordable encoding complexity that is rate independent. For such a codebook, an optimal search, and the indexing of the codevectors are addressed. Performance close to finite-dimensional random coding is achieved, while the complexity is not far from scalar coding. The structure of the codebook balances between complexity and performance. In comparison to some other methods, the advantage of the proposed scheme is more distinct for correlated sources.

We address speech coding architectures, suited for channels such as packetized networks. We adopt the analysis-by-synthesis paradigm. To achieve operability at a continuum of rates, all explicit perceptual filtering in the codebook search is eliminated, and quantization is performed with lattice quantizers. In this particular implementation, rate-distortion performance is improved by entropy coding of the lattice indices. The result is a competitive coder applicable to both speech and music, where subjective as well as objective performance scales well with the rate of the coder.

Recent advances in spectral coding for speech suggest Gaussian mixture modeling (GMM) as a tool for designing close to optimal, single stage quantizers. This combined with companding techniques promises to give flexible, no-memory methods that are essentially rate universal. We present methods for designing high performance companding structures based on a random union of Z-lattices. With this choice of basic lattice, VQ search is brought to the level of scalar quantization. OUT results show performance that clearly outperforms conventional VQ companding. OUT results are discussed in terms of the shape of the Vornoi cells of the overall quantizer. We apply the overall procedure to predictive spectrum quantization in two scenarios in terms of two audio bandwidths. For 16 kHz sampling we reach a spectral distortion (SD) of 1 at 33 bits per frame.