Contemporary text-to-speech solutions for accessibility applications can typically be classified into two categories: (i) device-based statistical parametric speech synthesis (SPSS) or unit selection (USEL) and (ii) cloud-based neural TTS. SPSS and USEL offer low latency and low disk footprint at the expense of naturalness and audio quality. Cloud-based neural TTS systems provide significantly better audio quality and naturalness but regress in terms of latency and responsiveness, rendering these impractical for real-world applications. More recently, neural TTS models were made de-ployable to run on handheld devices. Nevertheless, latency remains higher than SPSS and USEL, while disk footprint prohibits pre-installation for multiple voices at once. In this work, we describe a high-quality compact neural TTS system achieving latency on the order of 15 ms with low disk footprint. The proposed solution is capable of running on low-power devices.

Reverberation degrades signal quality and increases word error rates in automatic speech recognition (ASR). Reverberation suppression is, thus, a key component in listening enhancement devices and ASR front end. The weighted prediction error (WPE) is a prominent and effective method that gained popularity in recent ASR challenges. The need for iterative optimization in WPE leads to high computational cost and instabilities for short signals. Neural net (NN) supported WPE was proposed to alleviate these issues. However, NN training requires parallel data, i.e., reverberant and "clean" (direct sound plus early reflections) speech, which is not available in general. We show that the supporting network can be trained efficiently, without any supervision, using reverberant speech only. Consequently, adaptation to unseen environments is largely simplified. Network training involves the complete de-reverberation system and relies on complex-valued back propagation. The experimental validation confirms that, the proposed approach matches the performance of the method with parallel training data both in terms of perceptual quality and ASR word error rates.

Moderate and severe reverberation reduce speech intelligibility as a result of the overlap-masking effect, which constitutes the simultaneous observation of multiple delayed and attenuated copies of the speech signal. Recent progress has been made in ameliorating the degradation in intelligibility by adaptively controlling the signal gain as a function of both the signal statistics and the properties of the environment. While the intelligibility gain is at present small, it is significant and serves as a clear indication that reduced masking in non-stationary portions of the signal, under appropriate smoothness constraints, correlates well with an increase in intelligibility. A multi-modal modification framework is expected to improve performance further by i) introducing additional means to reduce masking in designated portions of the speech signal and ii) reducing signal distortion introduced by the use of a single modification modality. In particular we consider the combination of adaptive gain control and local time-warp dependent on the gain modification. A listening test shows that the proposed approach outperforms linear time warp. Objective validation results provide additional insight into the specifics of the proposed multi-modal modification approach.

Overlap-masking reduces speech intelligibility in reverberant environments. In contrast to additive noise, the masking signal depends on the past of the speech signal. An increase in output signal power is followed by an increase in reverberation power. Taking into consideration, the mechanics of reverberation is essential for the development of speech modifications that effectively increase intelligibility. This letter proposes a mathematical framework that optimizes the full-band signal power as a function of late reverberation power and the degree of signal nonstationarity. The prescribed signal gain is smoothed adaptively to suppress artifacts that may be introduced by rapid gain fluctuations due to frame-based processing. Compared to a reference method, it is shown that higher signal-to-late-reverberation ratio in nonstationary regions of the speech signal is achieved with less aggressive, on average, gain modification. A listening test with native English speakers on meaningful sentences under strong reverberation measured a consistent and significant improvement in intelligibility over natural speech and a reference method.

Speech intelligibility in noisy environments decreases with an increase in the noise power. We hypothesize that the differences of subsequent short-term spectra of speech, which we collectively refer to as the speech spectral dynamics, can be used to characterize speech intelligibility. We propose a distortion measure to characterize the deviation of the dynamics of the noisy modified speech from the dynamics of natural speech. Optimizing this distortion measure, we derive a parametric relationship between the signal band-power before and after modification. The parametric nature of the solution ensures adaptation to the noise level, the speech statistics and a penalty on the power gain. A multi-band speech modification system based on the single-band optimal solution is designed under a total signal power constraint and evaluated in selected noise conditions. The results indicate that the proposed approach compares favorably to a reference method based on optimizing a measure of the speech intelligibility index. Very low computational complexity and high intelligibility gain make this an attractive approach for speech modification in a wide range of application scenarios.

Speech is the most important communication modality for human interaction. Automatic speech recognition and speech synthesis have extended further the relevance of speech to man-machine interaction. Environment noise and various distortions, such as reverberation and speech processing artifacts, reduce the mutual information between the message modulated inthe clean speech and the message decoded from the observed signal. This degrades intelligibility and perceived quality, which are the two attributes associated with quality of service. An estimate of the state of these attributes provides important diagnostic information about the communication equipment and the environment. When the adverse effects occur at the presentation side, an objective measure of intelligibility facilitates speech signal modification for improved communication.The contributions of this thesis come from non-intrusive quality assessment and intelligibility-enhancing modification of speech. On the part of quality, the focus is on predictor design for limited training data. Paper A proposes a quality assessment model for bounded-support ratings that learns efficiently from a limited amount of training data, scales easily with the sampling frequency, and provides a platform for modeling variations in the individual subjective ratings. The predictive performance of the model for the mean of the subjective quality ratings compares favorably to the state-of-art in the field. Patterns in the spread of the individual ratings are captured in the feature space of the training data.Paper B focuses on enhancing predictive performance for the mean of the quality variable when the signal feature space is sparsely sampled by the training data. Using a Gaussian Processes framework, the deterministic signal-based feature set is augmented with a stochastic feature that is hypothesized to be jointly distributed with the target quality rating. An uncertainty propagation mechanism ensures that the variance of this feature is reflected in the prediction. The proposed architecture can take advantage of i) data that cannot be pooled due to subjective test protocol incompatibility and ii) models trained on data that are no longer available.With respect to intelligibility enhancement, a hierarchical perspective of the speech communication process, extended from foundational work in the field, is used in paper C to create a unified framework for method analysis and comparison. A high-level intelligibility measure related to the probability for correct recognition is derived using a hit-or-miss distortion criterion in the transcription domain. The measure is used to optimize two speech modifications at different levels of the message encoding hierarchy leading to significantly enhanced intelligibility in noise. The conceptual novelty of the method comes at the cost of higher complexity and the requirement for additional information including message transcription, sound segmentation, and a model of speech.Mapping the high-level measure to a lower level takes away the need for additional information and preserves asymptotically high-level optimality. Two methods are proposed to reduce degradation in the accuracy of the spectral dynamics due to additive noise. The focus of paper D is dynamics preservation in a range that is lower-bounded by an optimal band-power threshold. The performance of the method is competitive but allows for improvement in power efficiency. This issue is addressed in paper E which proposes and optimizes a distortion measure for spectral dynamics leading to a significant increase in intelligibility. Use of functional optimization techniques allows for families of solutions, among which are dynamic range compressors adaptive to the statistics of the speech and the noise.

Existing algorithms for improving speech intelligibility in a noisy environment generally focus on modifying the acoustic features of live, recorded or synthesized speech while preserving the phonetic composition (the message). In this paper, we present an algorithm for text-to-speech systems that operates at a higher level of abstraction, the message-level. We use a paraphrasing system to adjust the linguistic content of the intended message such that the speech intelligibility improves under noisy conditions. To distinguish the intelligibility among paraphrases, we use the numerical integration of a normalized log-likelihood function over different signal-to-noise conditions. Objective evaluation results show that the developed measure is able to distinguish the intelligibility among paraphrases. Results from subjective evaluation confirm the effectiveness of our objective measure.

An effective measure of speech intelligibility is the probability of correct recognition of the transmitted message. We propose a speech pre-enhancement method based on matching the recognized text to the text of the original message. The selected criterion is accurately approximated by the probability of the correct transcription given an estimate of the noisy speech features. In the presence of environment noise, and with a decrease in the signal-to-noise ratio, speech intelligibility declines. We implement a speech pre-enhancement system that optimizes the proposed criterion for the parameters of two distinct speech modification strategies under an energy-preservation constraint. The proposed method requires prior knowledge in the form of a transcription of the transmitted message and acoustic speech models from an automatic speech recognition system. Performance results from an open-set subjective intelligibility test indicate a significant improvement over natural speech and a reference system that optimizes a perceptual-distortion-based objective intelligibility measure. The computational complexity of the approach permits use in on-line applications.

The intelligibility of speech in adverse noise conditions can be improved by modifying the characteristics of the clean speech prior to its presentation. An effective and flexible paradigm is to select the modification by optimizing a measure of objective intelligibility. Here we apply this paradigm at the text level and optimize a measure related to the classification error probability in an automatic speech recognition system. The proposed method was applied to a simple but powerful band-energy modification mechanism under an energy preservation constraint. Subjective evaluation results provide a clear indication of a significant gain in subjective intelligibility. In contrast to existing methods, the proposed approach is not restricted to a particular modification strategy and treats the notion of optimality at a level closer to that of subjective intelligibility. The computational complexity of the method is sufficiently low to enable its use in on-line applications.

The development of objective speech quality measures generally involves fitting a model to subjective rating data. A typical data set comprises ratings generated by listening tests performed in different languages and across different laboratories. These factors as well as others, such as the sex and age of the talker, influence the subjective ratings and result in data heterogeneity. We use a linear hierarchical Bayes (HB) structure to account for heterogeneity. To make the structure effective, we develop a variational Bayesian inference for the linear HB structure that approximates not only the posterior over the model parameters, but also the model evidence. Using the approximate model evidence we are able to study and exploit the heterogeneity inducing factors in the Bayesian framework. The new approach yields a simple linear predictor with state-of-the-art predictive performance. Our experiments show that the new method compares favorably with systems based on more complex predictor structures such as ITU-T recommendation P.563, Bayesian MARS, and Gaussian processes.

Non-intrusive signal quality assessment in general, and its application to speech signal processing, in particular, builds extensively upon statistical regression models. Commonly, the raw preference scores used for fitting these models belong to a categorical scale. Averaging the scores over a number of test subjects results in smooth, close-to-continuous ratings, thus justifying the use of regression as opposed to classification models. A form of marginalization, averaging subjective ratings takes away useful information about the reliability of individual test points. Using a model tailored to the raw data achieves highly competitive performance in terms of conventional performance measures while providing the additional advantage of identifying the usability of individual test points. In this paper, we consider the application of discrete choice models to non-intrusive quality assessment of speech.

A fundamental challenge in the design of objective models for estimation of speech signal quality lies in the shortage of subjectively labelled databases. This problem is particularly relevant when developing quality assessment models for wide-band (16 kHz sampling rate) signals where databases are scarce. We explore the possibility for seamlessly integrating a quality prior in the form of a narrow-band quality estimate into the framework of a non-intrusive wide-band quality assessment algorithm. Experimental results confirm that the proposed approach can be used to improve performance over a baseline wide-band system without a narrow-band prior.

A Bayesian approach to non-intrusive quality assessment of narrow-band speech is presented. The speech features used to assess quality are the sample mean and variance of band-powers evaluated from the temporal envelope in the channels of an auditory filter-bank. Bayesian multivariate adaptive regression splines (BMARS) is used to map features into quality ratings. The proposed combination of features and regression method leads to a high performance quality assessment algorithm that learns efficiently from a small amount of training data and avoids overfitting. Use of the Bayesian approach also allows the derivation of credible intervals on the model predictions, which provide a quantitative measure of model confidence and can be used to identify the need for complementing the training databases.

A class of methods known as phase vocoders allows for implementing pitch shifting in the spectral domain. We extend the approach of shifting the isolated harmonies of the spectrum by introducing a ...