Neural network approaches to single-channel speech enhancement have received much recent attention. In particular, mask-based architectures have achieved significant performance improvements over conventional methods. This paper proposes a multiscale autoencoder (MSAE) for mask-based end-to-end neural network speech enhancement. The MSAE performs spectral decomposition of an input waveform within separate band-limited branches, each operating with a different rate and scale, to extract a sequence of multiscale embeddings. The proposed framework features intuitive parameterization of the autoencoder, including a flexible spectral band design based on the Constant-Q transform. Additionally, the MSAE is constructed entirely of differentiable operators, allowing it to be implemented within an end-to-end neural network, and be discriminatively trained. The MSAE draws motivation both from recent multiscale network topologies and from traditional multiresolution transforms in speech processing. Experimental results show the MSAE to provide clear performance benefits relative to conventional single-branch autoencoders. Additionally, the proposed framework is shown to outperform a variety of state-of-the-art enhancement systems, both in terms of objective speech quality metrics and automatic speech recognition accuracy.

In speaker verification, score calibration is employed to transform verification scores to log-likelihood ratios (LLRs) which are statistically interpretable. Conventional calibration techniques apply a global score transform. However, in condition-aware (CA) calibration, information conveying signal conditions is provided as input, allowing calibration to be adaptive. This paper explores a generative approach to condition-aware score calibration. It proposes a novel generative model for speaker verification trials, each which includes a trial score, a trial label, and the associated pair of speaker embeddings. Trials are assumed to be drawn from a discrete set of underlying signal conditions which are modeled as latent Categorical random variables, so that trial scores and speaker embeddings are drawn from condition-dependent distributions. An Expectation-Maximization (EM) Algorithm for parameter estimation of the proposed model is presented, which does not require condition labels and instead discovers relevant conditions in an unsupervised manner. The generative condition-aware (GCA) calibration transform is then derived as the log-likelihood ratio of a verification score given the observed pair of embeddings. Experimental results show the proposed approach to provide performance improvements on a variety of speaker verification tasks, outperforming static and condition-aware baseline calibration methods. GCA calibration is observed to improve the discriminative ability of the speaker verification system, as well as provide good calibration performance across a range of operating points. The benefits of the proposed method are observed for task-dependent models where signal conditions are known, for universal models which are robust across a range of conditions, and when facing unseen signal conditions.

This paperpresents a Bayesian framework for estimating a Probabilistic Linear Discriminant Analysis (PLDA) model in the presence of noisy labels. True class labels are interpreted as latent random variables, which are transmitted through a noisy channel, and received as observed speaker labels. The labeling process is modeled as a Discrete Memoryless Channel (DMC). PLDA hyperparameters are interpreted as random variables, and their joint posterior distribution is derived using mean-field Variational Bayes, allowing maximuma posteriori(MAP) estimates of the PLDA model parameters to be determined. The proposed solution, referred to as VB-MAP, is presented as a general framework, but is studied in the context of speaker verification, and a variety of use cases are discussed. Specifically, VB-MAP can be used for PLDA estimation with unreliable labels, unsupervised PLDA estimation, and to infer the reliability of a PLDA training set. Experimental results show the proposed approach to provide significant performance improvements on a variety of NIST Speaker Recognition Evaluation (SRE) tasks, both for data sets with simulated mislabels, and for data sets with naturally occurring missing or unreliable labels.

Future wearable technology may provide for enhanced communication in noisy environments and for the ability to pick out a single talker of interest in a crowded room simply by the listener shifting their attentional focus. Such a system relies on two components, speaker separation and decoding the listener's attention to acoustic streams in the environment. To address the former, we present a system for joint speaker separation and noise suppression, referred to as the Binaural Enhancement via Attention Masking Network (BEAMNET). The BEAMNET system is an end-to-end neural network architecture based on self-attention. Binaural input waveforms are mapped to a joint embedding space via a learned encoder, and separate multiplicative masking mechanisms are included for noise suppression and speaker separation. Pairs of output binaural waveforms are then synthesized using learned decoders, each capturing a separated speaker while maintaining spatial cues. A key contribution of BEAMNET is that the architecture contains a separation path, an enhancement path, and an autoencoder path. This paper proposes a novel loss function which simultaneously trains these paths, so that disabling the masking mechanisms during inference causes BEAMNET to reconstruct the input speech signals. This allows dynamic control of the level of suppression applied by BEAMNET via a minimum gain level, which is not possible in other state-of-the-art approaches to end-to-end speaker separation. This paper also proposes a perceptually-motivated waveform distance measure. Using objective speech quality metrics, the proposed system is demonstrated to perform well at separating two equal-energy talkers, even in high levels of background noise. Subjective testing shows an improvement in speech intelligibility across a range of noise levels, for signals with artificially added head-related transfer functions and background noise. Finally, when used as part of an auditory attention decoder (AAD) system using existing electroencephalogram (EEG) data, BEAMNET is found to maintain the decoding accuracy achieved with ideal speaker separation, even in severe acoustic conditions. These results suggest that this enhancement system is highly effective at decoding auditory attention in realistic noise environments, and could possibly lead to improved speech perception in a cognitively controlled hearing aid.

This paper proposes the Speech Enhancement via Attention Masking Network (SEAMNET), a neural network-based end-to-end single-channel speech enhancement system designed for joint suppression of noise and reverberation. It formalizes an end-to-end network architecture, referred to as b-Net, which accomplishes noise suppression through attention masking in a learned embedding space. A key contribution of SEAMNET is that the b-Net architecture contains both an enhancement and an autoencoder path. This paper proposes a novel loss function which simultaneously trains both the enhancement and the autoencoder paths, so that disabling the masking mechanism during inference causes SEAMNET to reconstruct the input speech signal. This allows dynamic control of the level of suppression applied by SEAMNET via a minimum gain level, which is not possible in other state-of-the-art approaches to end-to-end speech enhancement. This paper also proposes a perceptually-motivated waveform distance measure. In addition to the b-Net architecture, this paper proposes a novel method for designing target waveforms for network training, so that joint suppression of additive noise and reverberation can be performed by an end-to-end enhancement system, which has not been previously possible. Experimental results show the SEAMNET system to outperform a variety of state-of-the-art baselines systems, both in terms of objective speech quality measures and subjective listening tests. Finally, this paper draws parallels between SEAMNET and conventional statistical model-based enhancement approaches, offering interpretability of many network components.

This paper proposes a method for Bayesian estimation of probabilistic linear discriminant analysis (PLDA) when training labels are noisy. Label errors can be expected during e.g. large or distributed data collections, or for crowd-sourced data labeling. By interpreting true labels as latent random variables, the observed labels are modeled as outputs of a discrete memoryless channel, and the maximum a posteriori (MAP) estimate of the PLDA model is derived via Variational Bayes. The proposed framework can be used for PLDA estimation, PLDA domain adaptation, or to infer the reliability of a PLDA training list. Although presented as a general method, the paper discusses specific applications for speaker verification. When applied to the Speakers in the Wild (SITW) Task, the proposed method achieves graceful performance degradation when label errors are introduced into the training or domain adaptation lists. When applied to the NIST 2018 Speaker Recognition Evaluation (SRE18) Task, which includes adaptation data with noisy speaker labels, the proposed technique provides performance improvements relative to unsupervised domain adaptation.

Abstract : AbstractThis paper addresses speaker verification domain adaptationwith inadequate in-domain data. Specifically, we explore thecases where in-domain data sets do not include speaker labels,contain speakers with few samples, or contain speakers withlow channel diversity. Existing domain adaptation methods arereviewed, and their shortcomings are discussed. We derive anunsupervised version of fully Bayesian adaptation which reducesthe reliance on rich in-domain data. When applied todomain adaptation with inadequate in-domain data, the proposedapproach yields competitive results when the samples perspeaker are reduced, and outperforms existing supervised methodswhen the channel diversity is low, even without requiringspeaker labels. These results are validated on the NIST SRE16,which uses a highly inadequate in-domain data set.

This paper presents a framework for fully Bayesian speaker comparison of i-vectors. By generalizing the train/test paradigm, we derive an analytic expression for the speaker comparison log-likelihood ratio (LLR), as well as solutions for model training and Bayesian scoring. This framework is useful for enrollment sets of any size. For the specific case of single-cut enrollment, it is shown to be mathematically equivalent to probabilistic linear discriminant analysis (PLDA). Additionally, we present discriminative training of model hyper-parameters by minimizing the total cross entropy between LLRs and class labels. When applied to speaker recognition, significant performance gains are observed for various NIST SRE 2010 extended evaluation tasks.

In this paper we propose fully Bayesian speaker comparison of supervectors, which we refer to as SV-BSC, as a method for estimating whether a test cut was generated by the same speaker as an enrollment set. We derive the SV-BSC log-likelihood ratio of same-speaker to different-speaker hypotheses, and present solutions for model training and Bayesian scoring. We then show that if speaker and channel variability are assumed to inhabit a total variability subspace, SV-BSC scoring reduces to a form which requires only low-computation subspace operations. Finally, we show that common speaker recognition techniques such as Joint Factor Analysis (JFA) and i-vector Probabilistic Linear Discriminant Analysis (PLDA) are approximations to this full solution under certain additional assumptions. Experiments on the NIST 2010 SRE show SV-BSC to outperform a PLDA system.

We propose a method for spectral enhancement of reverberant speech based on inverting the modulation transfer function (MTF). Using all-pole models of modulation spectra allows the linear prediction inverse MTF (LP-IMTF) filter to exhibit a smooth frequency response, and allows it to be implemented as a low-order IIR filter in the modulation envelope domain. The proposed filter adapts to current acoustic conditions without relying on explicit information regarding reverberation time. Additionally, the LP-IMTF framework allows for estimation of useful side information, such as local signal-to-reverberation ratios and band-specific reverberation times. As example applications, the LP-IMTF system is applied to enhancement and speaker recognition of reverberant speech, and significant performance improvements are achieved.

This paper presents a family of log-spectral amplitude (LSA) estimators for speech enhancement. Generalized Gamma distributed (GGD) priors are assumed for speech short-time spectral amplitudes (STSAs), providing mathematical flexibility in capturing the statistical behavior of speech. Although solutions are not obtainable in closed-form, estimators are expressed as limits, and can be efficiently approximated. When applied to the Noizeus database [1], proposed estimators are shown to provide improvements in segmental signal-to-noise ratio (SSNR) and COSH distance [2], relative to the LSA estimator proposed by Ephraim and Malah [3].

This paper proposes a new statistical model-based likelihood ratio test (LRT) VAD to obtain reliable speech / non-speech decisions. In the proposed method, the likelihood ratio (LR) is calculated differently for voiced frames, as opposed to unvoiced frames: only DFT bins containing harmonic spectral peaks are selected for LR computation. To evaluate the new VAD's effectiveness in improving the noise-robustness of ASR, its decisions are applied to pre-processing techniques such as non-linear spectral subtraction, minimum mean square error short-time spectral amplitude estimator, and frame dropping. From the ASR experiments conducted on the Aurora2 database, the proposed harmonic frequency-based LRTs give better results than conventional LRT-based VADs and the standard G.729B and ETSI AMR VADs.

In this paper, we present efficient HMM-based techniques for estimating missing features. By assuming speech features to be observations of hidden Markov processes, we derive a minimum mean-square error (MMSE) solution. We increase the computational efficiency of HMM-based methods by downsampling underlying Markov models, and by enforcing symmetry in transitional probability matrices. When applied to features generally utilized in parametric speech coding, namely line spectral frequencies (LSFs), the proposed methods provide significant improvement over the baseline repetition scheme, in terms of Itakura-Saito distortion and peak SNR.

In this letter, we present a statistical approach to Mel-domain mask estimation for missing feature (MF)-based automatic speech recognition (ASR). Mel-domain time-frequency masks are of interest, since MF systems have been shown successful in that domain. Time- and channel-specific reliability measures are derived as posterior probabilities of active speech using a 2-state speech model. Since closed form distributions for Mel-domain spectra do not exist, they are instead modeled as χ2processes with empirically-determined degrees of freedom. Additionally, we present HMM-based decoding to exploit temporal correlation of spectral speech data. The proposed mask estimation algorithm is integrated with an example MF-based ASR front-end from, and is shown to outperform the spectral subtraction (SS)-based method from in terms of word-accuracy, when applied to the Aurora-2 database.

This paper presents a framework for efficient HMM-based estimation of unreliable spectrographic speech data. It discusses the role of hidden Markov models (HMMs) during minimum mean-square error (MMSE) spectral reconstruction. We develop novel HMM-based reconstruction algorithms which exploit intra-channel (across-time) correlation and/or inter-channel (across-frequency) correlation. For the sake of computational efficiency, this paper utilizes approximations to HMM-based decoding methods by developing models constructed from lower resolution quantizers. State configurations for lower resolution models are obtained through a tree-structured mapping of quantizer centroids, and model parameters are adapted accordingly. HMM downsampling avoids expensive retraining of models, and eliminates unnecessary memory requirements. Explicit general formulae are presented for the adaptation of steady-state and transitional statistics. Adaptation of observation statistics are derived from stochastic models of noise spectral magnitude estimation accuracies. The proposed estimation methods are applied in combination with oracle masks, which provide an upper performance bound, as well as masks derived from speech presence probability, which represent a more realistic scenario. Both methods are shown to boost noise robust recognition accuracies significantly relative to the Mel-frequency cepstral coefficient (MFCC) baseline system. Furthermore, HMM downsampling greatly reduces the complexity of the HMM-based reconstruction method while negligibly affecting results.

In previous work, a speech enhancement algorithm based on phase opponency and a periodicity measure (MPO-APP) was developed for speech recognition. Axiomatic thresholds were used in the MPO-APP regardless of the signal-to-noise ratio (SNR) of the corrupted speech or any characterization of the noise. The current work developed an algorithm for adjusting the threshold in the MPO-APP based on the SNR and whether the speech signal is clean, corrupted by aperiodic noise or corrupted with noise with periodic components. In addition, variable frame rate (VFR) analysis has been incorporated so that dynamic regions in the speech signal are more heavily sampled than steady-state regions. The result is a 2-stage algorithm that gives superior performance to the previous MPO-APP, and to several other state-of-the-art speech enhancement algorithms. Index Terms: Speech enhancement, robust speech recognition, SNR estimation, variable frame rate analysis, phase opponency.

In this letter, we propose a novel algorithm for reconstructing unreliable spectrographic data, a method applicable to missing feature-based automatic speech recognition (ASR). We provide quantitative analysis illustrating the high compressibility of spectrographic speech data. The existence of sparse representations for spectrographic data motivates the spectral reconstruction solution to be posed as an optimization problem minimizing the$\ell_{1}$-norm. When applied to the Aurora-2 database, the proposed missing feature estimation algorithm is shown to provide significant improvements in recognition accuracy relative to the baseline MFCC system. Even without an oracle mask, performance approaches that of the ETSI advanced front end (AFE), with less complexity.

We present the Networked InfoMechanical System for Planar Translation, which is a novel two-degree-of-freedom (2-DOF) cable-driven robot with self-calibration and online drift-correction capabilities. This system is intended for actuated sensing applications in aquatic environments. The actuation redundancy resulting from in-plane translation driven by four cables results in an infinite set of tension distributions, thus requiring real-time computation of optimal tension distributions. To this end, we have implemented a highly efficient, iterative linear programming solver, which requires a very small number of iterations to converge to the optimal value. In addition, two novel self-calibration methods have been developed that leverage the robot's actuation redundancy. The first uses an incremental displacement, or jitter method, whereas the second uses variations in cable tensions to determine end-effector location. We also propose a novel least-squares drift-detection algorithm, which enables the robot to detect long-term drift. Combined with self-calibration capabilities, this drift-monitoring algorithm enables long-term autonomous operation. To verify the performance of our algorithms, we have performed extensive experiments in simulation and on a real system.

This paper proposes an efficient approximation of the forward-backward (FB) algorithm, for the purpose of estimating missing features, based on downsampling statistical models. The paper discusses the role of hidden Markov models (HMMs) in the estimation process, and presents an approximation to the FB method by developing HMMs based on lower resolution quantizers, which are obtained through a tree-structure mapping of quantizer centroids. To illustrate the effectiveness of the proposed method, we apply it to the problem of error concealment in remote speech recognition, using the Aurora-2 database. The FB approximation provides comparable word recognition accuracy results relative to the standard FB method, while reducing the computational load by a large factor (> 250 in this case).

This paper presents a novel approach for reconstructing unreliable spectral components, which utilizes HMM-based missing feature algorithms, and applies them to noise robust speech recognition. The proposed technique uses the forwardbackward algorithm to estimate corrupt spectrographic data based on nearby reliable features, noisy observations, and on an underlying statistical model. The estimation process can be applied based on intra-channel information, intra-feature information, or a combination of both. The overall system is shown to provide vast improvements for the Consonant Challenge Database [1], for both MFCCs and PLP features, when using an oracle mask. Moreover, through downsampling of statistical models [2], the required complexity of the system is greatly reduced with negligible effects on results.

This paper proposes a novel low-complexity lip contour model for high-level optic feature extraction in noise-robust audiovisual (AV) automatic speech recognition systems. The model is based on weighted least-squares parabolic fitting of the upper and lower lip contours, does not require the assumption of symmetry across the horizontal axis of the mouth, and is therefore realistic. The proposed model does not depend on the accurate estimation of specific facial points, as do other high-level models. Also, we present a novel low-complexity algorithm for speaker normalization of the optic information stream, which is compatible with the proposed model and does not require parameter training. The use of the proposed model with speaker normalization results in noise robustness improvement in AV isolated-word recognition relative to using the baseline high-level model.

We propose a novel packetization and variable bitrate compression scheme for DSR source coding, based on the Group of Pictures concept from video coding. The proposed algorithm simultaneously packetizes and further compresses source coded features using the high interframe correlation of speech, and is compatible with a variety of VQ-based DSR source coders. The algorithm approximates vector quantizers as Markov Chains, and empirically trains the corresponding probability parameters. Feature frames are then compressed as I-frames, P-frames, or B-frames, using Huffman tables. The proposed scheme can perform lossless compression, but is also robust to lossy compression through VQ pruning or frame puncturing. To illustrate its effectiveness, we applied the proposed algorithm to the ETSI DSR source coder. The algorithm provided compression rates of up to 31.60% with negligible recognition accuracy degradation, and rates of up to 71.15% with performance degradation under 1.0%.

We propose a novel rate allocation algorithm for multiuser speech communication systems based on bargaining theory. Specifically, we apply the generalized Kalai-Smorodinsky bargaining solution since it allows varying bargaining powers to match the dynamic nature of speech signals. We propose a novel method to derive bargaining powers based on the short-time energy of the input speech signals, and subsequently allocate rates accordingly to the users. An important merit of the proposed framework is that it is general and can be applicable for resource allocation across a variety of multirate speech coders, and it is robust to a variety of speech quality metrics. The proposed system is also shown to involve a quick and low-complexity training process. We generalize the algorithm to scenarios in which users have unequally weighted priorities. These scenarios might arise in emergency situations, in which certain users are more important than others. The proposed rate allocation system is shown to increase the utility measures for both the Itakura and segmental signal-to-noise ratio (SNR) functions relative to the baseline system that performs uniform rate allocation. Additionally, although the instantaneous bitrate resolution of the speech encoder is not changed, the proposed system is shown to increase the short-time average bitrate resolution, and therefore provides a greater number of operational rate modes for the network