We propose a real-time speaker-independent speech-to-facial animation system that predicts lip and jaw movements on a reference face for audio speech taken from any speaker. Our approach is motivated by two key observations; 1) Speakerindependent facial animation can be generated from phoneme labels, but to perform this automatically a speech recogniser is needed which, due to contextual look-ahead, introduces too much time lag. 2) Audio-driven speech animation can be performed in real-time but requires large, multi-speaker audio-visual speech datasets of which there are few. We adopt a novel threestage training procedure that leverages the advantages of each approach. First we train aphoneme-to-visual speech model from a large single-speaker audio-visual dataset. Next, we use this model to generate the synthetic visual component of a large multi-speaker audio dataset of which the video is not available. Finally, we learn anaudio-to-visual speech mapping using the synthetic visual features as the target. Furthermore, we increase the realism of the predicted facial animation by introducing two perceptually-based loss functions that aim to improve mouth closures and openings. The proposed method and loss functions are evaluated objectively using mean square error, global variance and a new metric that measures the extent of mouth opening. Subjective tests show that our approach produces facial animation comparable to those produced from phoneme sequences and that improved mouth closures, particularly for bilabial closures, are achieved.

The aim of this paper is to examine denoising autoencoders (DAEs) for improving the detection of right whales recorded in harsh marine environments. Passive acoustic recordings are taken from autonomous surface vehicles (ASVs) and are subject to noise from sources such as shipping and offshore construction. To mitigate the noise we apply DAEs and consider how best to train the classifier by augmenting clean training data with examples contaminated by noise. Evaluations find that the DAE improves detection accuracy and is particularly effective when the classifier is trained on data that has itself been denoised rather than using a clean model. Further, testing on unseen noises is also effective particularly for noises that exhibit similar character to noises seen in training.

Machine learning has previously been applied successfully to speech-driven facial animation. To account for carry-over and anticipatory coarticulation a common approach is to predict the facial pose using a symmetric window of acoustic speech that includes both past and future context. Using future context limits this approach for animating the faces of characters in real-time and networked applications, such as online gaming. An acceptable latency for conversational speech is 200ms and typically network transmission times will consume a significant part of this. Consequently, we consider asymmetric windows by investigating the extent to which decreasing the future context effects the quality of predicted animation using both deep neural networks (DNNs) and bi-directional LSTM recurrent neural networks (BiLSTMs). Specifically we investigate future contexts from 170ms (fully-symmetric) to 0ms (fullyasymmetric …

This paper examines whether visual speech information can be effective within audio-masking-based speaker separation to improve the quality and intelligibility of the target speech. Two visual-only methods of generating an audio mask for speaker separation are first developed. These use a deep neural network to map the visual speech features to an audio feature space from which both visually derived binary masks and visually derived ratio masks are estimated, before application to the speech mixture. Second, an audio ratio masking method forms a baseline approach for speaker separation which is extended to exploit visual speech information to form audio-visual ratio masks. Speech quality and intelligibility tests are carried out on the visual-only, audio-only, and audio-visual masking methods of speaker separation at mixing levels from -10 to +10 dB. These reveal substantial improvements in the target speech when applying the visual-only and audio-only masks, but with highest performance occurring when combining audio and visual information to create the audio-visual masks.

This work proposes and compares perceptually motivated loss functions for deep learning based binary mask estimation for speech separation. Previous loss functions have focused on maximising classification accuracy of mask estimation but we now propose loss functions that aim to maximise the hit mi- nus false-alarm (HIT-FA) rate which is known to correlate more closely to speech intelligibility. The baseline loss function is bi- nary cross-entropy (CE), a standard loss function used in binary mask estimation, which maximises classification accuracy. We propose first a loss function that maximises the HIT-FA rate in- stead of classification accuracy. We then propose a second loss function that is a hybrid between CE and HIT-FA, providing a balance between classification accuracy and HIT-FA rate. Eval- uations of the perceptually motivated loss functions with the GRID database show improvements to HIT-FA rate and ESTOI across babble and factory noises. Further tests then explore ap- plication of the perceptually motivated loss functions to a larger vocabulary dataset.

This paper is concerned with generating intelligible audio speech from a video of a person talking. Regression and classification methods are proposed first to estimate static spectral envelope features from active appearance model visual features. Two further methods are then developed to incorporate temporal information into the prediction: A feature-level method using multiple frames and a model-level method based on recurrent neural networks. Speech excitation information is not available from the visual signal, so methods to artificially generate aperiodicity and fundamental frequency are developed. These are combined within the STRAIGHT vocoder to produce a speech signal. The various systems are optimized through objective tests before applying subjective intelligibility tests that determine a word accuracy of 85% from a set of human listeners on the GRID audio-visual speech database. This compares favorably with a previous regression-based system that serves as a baseline, which achieved a word accuracy of 33%.

This work proposes a method of speech enhancement that uses a network of HMMs to first decode noisy speech and to then synthesise a set of features that enables a clean speech signal to be reconstructed. Different choices of acoustic model (whole-word, monophone and triphone) and grammars (highly constrained to no constraints) are considered and the effects of introducing or relaxing acoustic and grammar constraints investigated. For robust operation in noisy conditions it is necessary for the HMMs to model noisy speech and consequently noise adaptation is investigated along with its effect on the reconstructed speech. Speech quality and intelligibility analysis find triphone models with no grammar, combined with noise adaptation, gives highest performance that outperforms conventional methods of enhancement at low signal-to-noise ratios.

We study the problem of mapping from acoustic to visual speech with the goal of generating accurate, perceptually natural speech animation automatically from an audio speech signal. We present a sliding window deep neural network that learns a mapping from a window of acoustic features to a window of visual features from a large audio-visual speech dataset. Overlapping visual predictions are averaged to generate continuous, smoothly varying speech animation. We outperform a baseline HMM inversion approach in both objective and subjective evaluations and perform a thorough analysis of our results.

This paper examines methods to improve visual speech synthesis from a text input using a deep neural network (DNN). Two representations of the input text are considered, namely into phoneme sequences or dynamic viseme sequences. From these sequences, contextual features are extracted that include information at varying linguistic levels, from frame level down to the utterance level. These are extracted from a broad sliding window that captures context and produces features that are input into the DNN to estimate visual features. Experiments first compare the accuracy of these visual features against an HMM baseline method which establishes that both the phoneme and dynamic viseme systems perform better with best performance obtained by a combined phoneme-dynamic viseme system. An investigation into the features then reveals the importance of the frame level information which is able to avoid discontinuities in the visual feature sequence and produces a smooth and realistic output.

This work describes an investigation into the feasibility of producing intelligible audio speech from only visual speech fea- tures. The proposed method aims to estimate a spectral enve- lope from visual features which is then combined with an arti- ficial excitation signal and used within a model of speech pro- duction to reconstruct an audio signal. Different combinations of audio and visual features are considered, along with both a statistical method of estimation and a deep neural network. The intelligibility of the reconstructed audio speech is measured by human listeners, and then compared to the intelligibility of the video signal only and when combined with the reconstructed audio.

This work proposes a method to exploit both audio and vi- sual speech information to extract a target speaker from a mix- ture of competing speakers. The work begins by taking an ef- fective audio-only method of speaker separation, namely the soft mask method, and modifying its operation to allow visual speech information to improve the separation process. The au- dio input is taken from a single channel and includes the mix- ture of speakers, where as a separate set of visual features are extracted from each speaker. This allows modification of the separation process to include not only the audio speech but also visual speech from each speaker in the mixture. Experimen- tal results are presented that compare the proposed audio-visual speaker separation with audio-only and visual-only methods us- ing both speech quality and speech intelligibility metrics.

A study is presented on how well objective measures of speech quality and intelligibility can predict the subjective in- telligibility of speech that has undergone spectral envelope smoothing and simplification of its excitation. Speech modi- fications are made by resynthesising speech that has been spec- trally smoothed. Objective measures are applied to the mod- ified speech and include measures of speech quality, signal- to-noise ratio and intelligibility, as well as proposing the nor- malised frequency-weighted spectral distortion (NFD) measure. The measures are compared to subjective intelligibility scores where it is found that several have high correlation (|r| ≥ 0.7), with NFD achieving the highest correlation (r = −0.81)

This paper proposes a method of speech enhancement where a clean speech signal is reconstructed from a sinusoidal model of speech production and a set of acoustic speech features. The acoustic features are estimated from noisy speech and comprise, for each frame, a voicing classification (voiced, unvoiced or non-speech), fundamental frequency (for voiced frames) and spectral envelope. Rather than using different algorithms to estimate each parameter, a single statistical model is developed. This comprises a set of acoustic models and has similarity to the acoustic modelling used in speech recognition. This allows noise and speaker adaptation to be applied to acoustic feature estimation to improve robustness. Objective and subjective tests compare reconstruction-based enhancement with other methods of enhancement and show the proposed method to be highly effective at removing noise.

This paper proposes using a hidden Markov model (HMM) to model a speech signal in terms of its speech class (voiced, unvoiced and nonspeech) and for voiced speech its fundamental frequency. States of the HMM represent unvoiced speech and nonspeech with multiple voiced states that model different fundamental frequencies. The transition matrix of the HMM models temporal changes in speech class and the time-varying fundamental frequency contour. The model is then applied to voicing and fundamental frequency estimation by extracting acoustic features from a speech signal and then applying Viterbi decoding. Experimental results are presented that investigate the estimation accuracy of the proposed system and a comparison is made against conventional methods.

This paper proposes a set of higher-order modified moments for estimation of the fundamental frequency of speech and explores the impact of the speech window length on pitch estimation error. The pitch extraction methods are evaluated in a range of noise types and SNRs. For calculation of errors, pitch reference values are calculated from manually-corrected estimates of the periods obtained from laryngograph signals. The results obtained for the 3rd and 4th order modified moment compare well with methods based on correlation and magnitude difference criteria and the YIN method; with improved pitch accuracy and less occurrence of large errors

The aim of this work is to examine whether visual speech information can be used to enhance audio speech that has been contaminated by noise. First, an analysis of audio and visual speech features is made, which identifies the pair with highest audio-visual correlation. The study also reveals that higher audio-visual correlation exists within individual phoneme sounds rather than globally across all speech. This correlation is exploited in the proposal of a visually derived Wiener filter that obtains clean speech and noise power spectrum statistics from visual speech features. Clean speech statistics are estimated from visual features using a maximum a posteriori framework that is integrated within the states of a network of hidden Markov models to provide phoneme localization. Noise statistics are obtained through a novel audio-visual voice activity detector which utilizes visual speech features to make robust speech/nonspeech classifications. The effectiveness of the visually derived Wiener filter is evaluated subjectively and objectively and is compared with three different audio-only enhancement methods over a range of signal-to-noise ratios.

This paper examines the effect of applying noise compensation to acoustic speech feature prediction from noisy mel-frequency cepstral coefficient (MFCC) vectors within a distributed speech recognition architecture. An acoustic speech feature (comprising fundamental frequency, formant frequencies, speech/nonspeech classification, and voicing classification) is predicted from an MFCC vector in a maximum a posteriori (MAP) framework using phoneme-specific or global models of speech. The effect of noise is considered and three different noise compensation methods, that have been successful in robust speech recognition, are integrated within the MAP framework. Experiments show that noise compensation can be applied successfully to prediction with best performance given by a model adaptation method that performs only slightly worse than matched training and testing. Further experiments consider application of the predicted acoustic features to speech reconstruction. A series of human listening tests show that the predicted features are sufficient for speech reconstruction and that noise compensation improves speech quality in noisy conditions.

This paper proposes a set of higher-order modified moments as alternative objective criteria for pitch extraction and explores the impact of the speech window length on pitch estimation error. To obtain the Kthorder modified moment, each speech frame is split into a positive-valued signal and a negative-valued signal. The magnitudes of the Kthorder moments for the positive and the negative valued signals are obtained and combined. The proposed objective criteria form a relatively sharp peak around the true pitch value compared to the correlation function. For calculation of errors, pitch reference (`ground truth') values are calculated from manually-corrected estimates of the periods obtained from laryngograph signals. The results obtained for the third order modified moment are compared with the results for correlation and magnitude difference criteria and the YIN method. The modified moments provide improved pitch accuracy with less occurrence of large errors (e.g. half or double pitch estimation errors).

This work presents a novel approach to speech enhancement by exploiting the bimodality of speech and the correlation that exists between audio and visual speech features. For speech enhancement, a visually-derived Wiener filter is developed. This obtains clean speech statistics from visual features by modelling their joint density and making a maximum a posteriori estimate of clean audio from visual speech features. Noise statistics for the Wiener filter utilise an audio-visual voice activity detector which classifies input audio as speech or nonspeech, enabling a noise model to be updated. Analysis shows estimation of speech and noise statistics to be effective with human listening tests measuring the effectiveness of the resulting Wiener filter. Index Terms: Audio-visual, speech enhancement, Wiener filter 1.

The aim of this work is to reconstruct clean speech solely from a stream of noise-contaminated MFCC vectors, as may be encountered in distributed speech recognition systems. Speech reconstruction is performed using the ETSI Aurora back-end speech reconstruction standard which requires MFCC vectors, fundamental frequency and voicing information. In this work, fundamental frequency and voicing are obtained using maximum a posteriori prediction from input MFCC vectors, thereby allowing speech reconstruction solely from a stream of MFCC vectors. Two different methods to improve prediction accuracy in noisy conditions are then developed. Experimental results first establish that improved fundamental frequency and voicing prediction is obtained when noise compensation is applied. A series of human listening tests are then used to analyse the reconstructed speech quality, which determine the effectiveness of noise compensation in terms of mean opinion scores.

This paper examines the effect of applying noise compensation to improve acoustic speech feature prediction from noise contaminated MFCC vectors, as may be encountered in distributed speech recognition (DSR). A brief review of maximum a posteriori prediction of acoustic speech features (voicing, fundamental and formant frequencies) from MFCC vectors is made. Two noise compensation methods are then applied; spectral subtraction and model adaptation. Spectral subtraction is used to filter noise from the received MFCC vectors, while model adaptation is applied to adapt the joint models of acoustic features and MFCCs to account for noise contamination. Experiments examine acoustic feature prediction accuracy in noise and results show that the two noise compensation methods significantly improve prediction accuracy in noise. The technique of model adaptation was found to be better than spectral subtraction and could restore performance close to that achieved in matched training and testing.

This work begins by examining the correlation between audio and visual speech features and reveals higher correlation to exist within individual phoneme sounds rather than globally across all speech. Utilising this correlation, a visually-derived Wiener filter is proposed in which clean power spectrum estimates are obtained from visual speech features. Two methods of extracting clean power spectrum estimates are made; first from a global estimate using a single Gaussian mixture model (GMM), and second from phoneme-specific estimates using a hidden Markov model (HMM)-GMM structure. Measurement of estimation accuracy reveals that the phoneme-specific (HMM-GMM) system leads to lower estimation errors than the global (GMM) system. Finally, the effectiveness of visually-derived Wiener filtering is examined.

This work develops a statistical framework to predict acoustic features (fundamental frequency, formant frequencies and voicing) from MFCC vectors. An analysis of correlation between acoustic features and MFCCs is made both globally across all speech and within phoneme classes, and also from speaker-independent and speaker-dependent speech. This leads to the development of both a global prediction method, using a Gaussian mixture model (GMM) to model the joint density of acoustic features and MFCCs, and a phoneme-specific prediction method using a combined hidden Markov model (HMM)-GMM. Prediction accuracy measurements show the phoneme-dependent HMM-GMM system to be more accurate which agrees with the correlation analysis. Results also show prediction to be more accurate from speaker-dependent speech which also agrees with the correlation analysis.

This work compares the accuracy of fundamental frequency and formant frequency estimation methods and maximum a posteriori (MAP) prediction from MFCC vectors with hand-corrected references. Five fundamental frequency estimation methods are compared to fundamental frequency prediction from MFCC vectors in both clean and noisy speech. Similarly, three formant frequency estimation and prediction methods are compared. An analysis of estimation and prediction accuracy shows that prediction from MFCCs provides the most accurate voicing classification across clean and noisy speech. On clean speech, fundamental frequency estimation outperforms prediction from MFCCs, but as noise increases the performance of prediction is significantly more robust than estimation. Formant frequency prediction is found to be more accurate than estimation in both clean and noisy speech. A subjective analysis of the estimation and prediction methods is also made by reconstructing speech from the acoustic features.

This work proposes a method for predicting the fundamental frequency and voicing of a frame of speech from its mel-frequency cepstral coefficient (MFCC) vector representation. This information is subsequently used to enable a speech signal to be reconstructed solely from a stream of MFCC vectors and has particular application in distributed speech recognition systems. Prediction is achieved by modeling the joint density of fundamental frequency and MFCCs. This is first modeled using a Gaussian mixture model (GMM) and then extended by using a set of hidden Markov models to link together a series of state-dependent GMMs. Prediction accuracy is measured on unconstrained speech input for both a speaker-dependent system and a speaker-independent system. A fundamental frequency prediction error of 3.06% is obtained on the speaker-dependent system in comparison to 8.27% on the speaker-independent system. On the speaker-dependent system 5.22% of frames have voicing errors compared to 8.82% on the speaker-independent system. Spectrogram analysis of reconstructed speech shows that highly intelligible speech is produced with the quality of the speaker-dependent speech being slightly higher owing to the more accurate fundamental frequency and voicing predictions

This work proposes a noise reduction system for use in high noise environments such as motor racing. First, an analysis of the audio data received from the driver reveals that the main noise sources are from the engine, airflow and tyres. These are found to relate to engine speed, road speed and throttle position information that is received in a data stream from the car's on-board computer. A two-stage noise compensation strategy is proposed which first suppresses engine harmonics using adaptive filtering with engine speed reference information taken from the data stream. Second, a maximum a posteriori (MAP) prediction of the tyre and airflow noise is made from data stream values and this is combined with spectral subtraction for noise suppression. Human listening tests reveal that both noise reduction stages lead to good improvements in the intelligibility of the speech with a comparative mean opinion score (CMOS) of +1.57 being obtained.

The aim of this work is to examine the correlation between audio and visual speech features. The motivation is to find visual features that can provide clean audio feature estimates which can be used for speech enhancement when the original audio signal is corrupted by noise. Two audio features (MFCCs and formants) and three visual features (active appearance model, 2-D DCT and cross-DCT) are considered with correlation measured using multiple linear regression. The correlation is then exploited through the development of a maximum a posteriori (MAP) prediction of audio features solely from the visual features. Experiments reveal that features representing broad spectral information have higher correlation to visual features than those representing finer spectral detail. The accuracy of prediction follows the results found in the correlation measurements. Index Terms: audio-visual speech, correlation, AAM, formants 1.

This paper describes how formant frequencies of voiced and unvoiced speech can be predicted from mel-frequency cepstral coefficients (MFCC) vectors using maximum a posteriori (MAP) estimation within a hidden Markov model (HMM) framework. Gaussian mixture models (GMMs) are used to model the local joint density of MFCCs and formant frequencies. More localised prediction is achieved by modelling speech using voiced, unvoiced and non-speech GMMs for every state of each model of a set of HMMs. To predict formant frequencies from a MFCC vector, first a prediction of the speech class (voiced, unvoiced or non-speech) is made. Formant frequencies are predicted from voiced and unvoiced speech using a MAP estimation made using the state-specific GMMs. This 'eHMM-GMM' prediction of speech class and formant frequencies was evaluated on a male 5000 word unconstrained large vocabulary speaker-independent database.

This paper addresses the problem of achieving robust distributed speech recognition in the presence of burst-like packet loss. To compensate for packet loss a number of techniques are investigated to provide estimates of lost vectors. Experimental results on both a connected digits task and a large vocabulary continuous speech recognition task show that simple methods, such as repetition, are not as effective as interpolation methods which are better able to preserve the dynamics of the feature vector stream. Best performance is given by maximum a-posteriori (MAP) estimation of lost vectors which utilizes statistics of the feature vector stream. At longer burst lengths the performance of these compensation techniques deteriorates as the temporal correlation in the received feature vector stream reduces. To compensate for this interleaving is proposed which aims to disperse bursts of loss into a series of unconnected smaller bursts. Results show substantial gains in accuracy, to almost that of the no loss condition, when interleaving is combined with estimation techniques, although this is at the expense of introducing delay. This leads to the proposal that, for a distributed speech recognition application, it is more beneficial to trade delay for accuracy rather than trading bit-rate for accuracy as in forward error correction schemes.

This work proposes a novel method of predicting formant frequencies from a stream of mel-frequency cepstral coefficients (MFCC) feature vectors. Prediction is based on modelling the joint density of MFCCs and formant frequencies using a Gaussian mixture model (GMM). Using this GMM and an input MFCC vector, two maximum a posteriori (MAP) prediction methods are developed. The first method predicts formants from the closest, in some sense, cluster to the input MFCC vector, while the second method takes a weighted contribution of formants predicted from all clusters. Experimental results are presented using the ETSI Aurora connected digit database and show that predicted formant frequencies are within 3.2% of reference formant frequencies.

The aim of this work is to improve distributed speech recognition accuracy in packet loss by considering the effect of loss on the temporal derivatives of the feature vector. Analysis of temporal derivatives reveals they suffer severe distortion when static vectors are lost in times of packet loss. The application of missing feature theory and soft-decoding techniques are considered for compensating against packet loss at the decoding stage of recognition. An extension to these methods is developed which considers the static, velocity and acceleration components separately. A series of confidence measures for the temporal derivatives is devised and applied within the soft-decoding framework. Experimental results on both a connected digit task and a large vocabulary task demonstrate significant increases in recognition accuracy under a range of packet loss conditions.

This paper proposes a method of predicting the formant frequencies of a frame of speech from its mel-frequency cepstral coefficient (MFCC) representation. Prediction is achieved through the creation of a Gaussian mixture model (GMM) which models the joint density of formant frequencies and MFCCs. Using this GMM and an input MFCC vector, a maximum a posteriori (MAP) prediction of the formant frequencies is generated. Formant prediction accuracy is evaluated on both a constrained vocabulary connected digits database and on a 5000 word large vocabulary database. Experiments first examine the accuracy of formant frequency prediction as the number of clusters in the GMM is varied with a best formant frequency prediction error of 3.72% being obtained. Secondly the effect of noise on formant prediction accuracy is examined. A fall in accuracy is observed with reducing signal-to-noise ratios, but by using a GMM matched to the noise conditions formant prediction accuracy is significantly improved.

This paper presents a combined packet loss compensation system for distributed speech recognition (DSR). Compensation is applied at three stages within the DSR process beginning with interleaving on the terminal device to reduce burst lengths in the received feature vector stream. On the receiver side estimation of missing vectors is applied to reconstruct the feature vector stream prior to recognition. Finally, the decoding process of the recogniser is modified to take into account the varying reliability of these estimated feature vectors. Experiments performed on both the Aurora connected digits task and the WSJCAM0 large vocabulary task show substantial gains in recognition accuracy across a range of packet loss conditions.

This work proposes a method to predict the fundamental frequency and voicing of a frame of speech from its MFCC representation. This has particular use in distributed speech recognition systems where the ability to predict fundamental frequency and voicing allows a time-domain speech signal to be reconstructed solely from the MFCC vectors. Prediction is achieved by modeling the joint density of MFCCs and fundamental frequency with a combined hidden Markov model-Gaussian mixture model (HMM-GMM) framework. Prediction results are presented on unconstrained speech using both a speaker-dependent database and a speaker-independent database. Spectrogram comparisons of the reconstructed and original speech are also made. The results show for the speaker-dependent task a percentage fundamental frequency prediction error of 3.1% is made while for the speaker-independent task this rises to 8.3%.

This paper presents a formant-tracking method for estimation of the time-varying trajectories of a linear prediction (LP) model of speech in noise. The main focus of this work is on the modelling of the non-stationary temporal trajectories of the formants of speech for improved LP model estimation in noise. The proposed approach provides a systematic framework for modelling the inter-frame correlation of speech parameters across successive frames, the intra-frame correlations are modelled by LP parameters. The formant-tracking LP model estimation is composed of two stages: (a) a pre-cleaning intra-frame spectral amplitude estimation stage where an initial estimate of the magnitude frequency response of the LP model of clean speech is obtained and (b) an inter-frame signal processing stage where formant classification and Kalman filters are combined to estimate the trajectory of formants. The effects of car and train noise on the observations and estimation of formants tracks are investigated. The average formant tracking errors at different signal to noise ratios (SNRs) are computed. The evaluation results demonstrate that after noise reduction and Kalman filtering the formant tracking errors are significantly reduced. 1.

An analysis into the effect of packet loss shows that a speech recogniser is able to tolerate large percentages of packet loss provided that burst lengths are relatively small. This leads to the analysis of three types of interleaver for distributing long bursts of packet loss into a series of shorter bursts. Cubic interpolation is then used to estimate lost feature vectors. Experimental results are presented for a range of channel conditions and demonstrate that interleaving offers significant increases in recognition accuracy under burst-like packet loss. Of the interleavers tested, decorrelated interleaving gives superior recognition performance and has the lowest delay. For example at a packet loss rate of 50% and average burst length 20 packets (40 vectors or 400ms) performance is increased from 49.6% with no compensation to 86% with interleaving and cubic interpolation.

The paper proposes a technique for reconstructing an acoustic speech signal solely from a stream of Mel-frequency cepstral coefficients (MFCCs). Previous speech reconstruction methods have required an additional pitch element, but this work proposes two maximum a posteriori (MAP) methods for predicting pitch from the MFCC vectors themselves. The first method is based on a Gaussian mixture model (GMM) while the second scheme utilises the temporal correlation available from a hidden Markov model (HMM) framework. A formal measurement of both frame classification accuracy and RMS pitch error shows that an HMM-based scheme with 5 clusters per state is able to classify correctly over 94% of frames and has an RMS pitch error of 3.1 Hz in comparison to a reference pitch. Informal listening tests and analysis of spectrograms reveals that speech reconstructed solely from the MFCC vectors is almost indistinguishable from that using the reference pitch.

This work compares the performance of three compensation methods for speech recognition in the presence of packet loss. Two methods, cubic interpolation and a novel maximum a posteriori (MAP) estimation, aim to restore the feature vector stream in the event of packet loss, while the third technique applies compensation in the decoding stage of recognition through missing feature theory. To improve performance in burst-like packet loss, interleaving is introduced to disperse bursts of loss. Experiments on the ETSI Aurora connected digit task show best performance to be given by a combination of missing feature theory and cubic interpolation. This raises performance from 50.3% to 69.8% at a packet loss rate of 50% and average burst length of 20 packets. Including interleaving further increases performance to over 76%.

The evaluation of packet loss compensation techniques for distributed speech recognition requires an effective model of packet loss that is capable of reproducing the burst-like occurrence of loss. Several models have been applied to this task and are based on two or three state Markov chains or Markov models. This work reviews these models in terms of their channel characteristics such as the probability of packet loss and average burst length. Validation of the models is made against both GSM error patterns and a wireless LAN channel which demonstrates effective simulation. A series of speech recognition tests show that similar performance is obtained on the real and simulated channels using the packet loss models. Finally a set of model parameters is presented which allows testing across a range of channel conditions. 1.

This work proposes a method of predicting pitch and voicing from mel-frequency cepstral coefficient (MFCC) vectors. Two maximum a posteriori (MAP) methods are considered. The first models the joint distribution of the MFCC vector and pitch using a Gaussian mixture model (GMM) while the second method also models the temporal correlation of the pitch contour using a combined hidden Markov model (HMM)-GMM framework. Monophone-based HMMs are connected together in the form of an unconstrained monophone grammar which enables pitch to be predicted from unconstrained speech input. Evaluation on 130,000 MFCC vectors reveals a voicing classification accuracy of over 92% and an RMS pitch error of 10Hz. The predicted pitch contour is also applied to MFCC-based speech reconstruction with the resultant speech almost indistinguishable from that reconstructed using a reference pitch. 1.

The paper presents a novel method for the low bit-rate compression of a feature vector stream with particular application to distributed speech recognition. The scheme operates by grouping feature vectors into non-overlapping blocks and applying a transformation to give a more compact matrix representation. Both Karhunen-Loeve and discrete cosine transforms are considered. Following transformation, higher-order columns of the matrix can be removed without loss in recognition performance. The number of bits allocated to the remaining elements in the matrix is determined automatically using a measure of their relative information content. Analysis of the amplitude distribution of the elements indicates that non-linear quantisation is more appropriate than linear quantisation. Comparative results, based on both spectral distortion and speech recognition accuracy, confirm this. Speech recognition tests using the ETSI Aurora database demonstrate that compression to bits rates of 2400 bps, 1200 bps and 800 bps has very little effect on recognition accuracy. For example at a bit rate of 1200 bps, recognition accuracy is 98.0% compared to 98.6% with no compression.

This paper extends the technique of speech reconstruction from MFCC by considering the effect of noisy speech. To reconstruct a clean speech signal from noise contaminated MFCC an estimate of the clean mel-filterbank vector is required together with a robust estimate of the pitch. This work applies spectral subtraction to the mel-filterbank vector (derived from noisy MFCC) to provide a clean speech spectral estimate. To obtain a reliable estimate of pitch a robust extraction technique is used. Spectrograms and informal listening tests reveal that a clean speech signal can be successfully reconstructed from the noisy MFCC. Pitch errors are shown to manifest themselves as artificial sounding bursts in the reconstructed speech signal. Incorrect estimates of the spectral envelope introduce periods of noise into the reconstructed speech.

In concatenative text-to-speech (TTS) synthesis systems unit selection aims to reduce the number of concatenation points in the synthesized speech and make concatenation joins as smooth as possible. This research considers synthesis of completely new utterances from non-uniform units, whereby the most appropriate units, according to acoustic and phonetic criteria, are selected from a myriad of similar speech database candidates. A Viterbi-style algorithm dynamically selects the most suitable database units from a large speech database by considering concatenation and target costs. Concatenation costs are derived from mel filter bank amplitudes, whereas target costs are considered in terms of the phonemic and phonetic properties of required units. Within subjects and between subjects ANOVA [9] evaluation of listeners' scores showed that the TTS system with this method of unit selection was preferred in 52% of test sentences.

Context-awareness is essential to the development of adaptive information systems. Environmental noise can provide a rich source of information about the current context. We describe our approach for automatically sensing and recognising noise from typical environments of daily life, such as office, car and city street. In this paper we present our hidden Markov model based noise classifier. We describe the architecture of the system, compare classification results from the system with human listening tests, and discuss open issues in environmental noise classification for mobile computing. 1.

The aim of this work is to improve the robustness of speech recognition systems operating in burst-like packet loss. First a set of highly artificial packet loss profiles are used to analyse their effect on both recognition performance and on the underlying feature vector stream. This indicates that the simple technique of vector repetition can make the recogniser robust to high percentages of packet loss, providing burst lengths are reasonably short. This leads to the proposal of interleaving the feature vector sequence, prior to packetisation, to disperse bursts of packet loss throughout the feature vector stream.

This paper proposes an integrated speech front-end for both speech recognition and speech reconstruction applications. Speech is first decomposed into a set of frequency bands by an auditory model. The output of this is then used to extract both robust pitch estimates and MFCC vectors. Initial tests used a 128 channel auditory model, but results show that this can be reduced significantly to between 23 and 32 channels. A detailed analysis of the pitch classification accuracy and the RMS pitch error shows the system to be more robust than both comb function and LPC-based pitch extraction. Speech recognition results show that the auditory-based cepstral coefficients give very similar performance to conventional MFCCs. Spectrograms and informal listening tests also reveal that speech reconstructed from the auditory-based cepstral coefficients and pitch has similar quality to that reconstructed from conventional MFCCs and pitch. 1.

A criterion function for selecting the optimal payload size in data transmission over a binary symmetrical communication channel is presented. The work assumes that the packets comprise a fixed size header and variable length payload. The conventional criterion function is based on the ratio between payload size and mean number of transmitted bits per packet, which includes re-transmissions of corrupted packets. Applying a logarithm to the inverse of this criterion function makes the resulting function very suitable for mathematical analysis. This also facilitates parameter sensitivity studies without recourse to numerical methods. As such, this helps to visualise and understand the problem of optimal payload length selection when teaching courses in signal processing for communications and multimedia.

This paper presents a comparative analysis of the processing stages involved in feature extraction for speech recognition. Feature extraction is considered as comprising three different processing stages; namely static feature extraction, normalisation and inclusion of temporal information. In each stage a comparison of techniques is made, both theoretically and in terms of their comparative performance. The analysis shows that while some techniques may appear significantly different, upon analysis the effect they have on the signal can be similar. Comparative studies include MFCC and PLP analysis, RASTA filtering and cepstral mean normalisation, and temporal derivatives and cepstral-time matrices. Experimental results, on an unconstrained monophone task, compare recognition performance using different front-end configurations.

This work presents a method of reconstructing a speech signal from a stream of MFCC vectors using a source-filter model of speech production. The MFCC vectors are used to provide an estimate of the vocal tract filter. This is achieved by inverting the MFCC vector back to a smoothed estimate of the magnitude spectrum. The Wiener- Khintchine theorem and linear predictive analysis transform this into an estimate of the vocal tract filter coefficients. The excitation signal is produced from a series of pitch pulses or white noise, depending on whether the speech is voiced or unvoiced. This pitch estimate forms an extra element of the feature vector. Listening tests reveal that the reconstructed speech is intelligible and of similar quality to a system based on LPC analysis of the original speech. Spectrograms of the MFCC-derived speech and the real speech are included which confirm the similarity.

The technique of distributed speech recognition (DSR) has recently become an interesting area of research. One of the main issues with DSR is the need to compress the feature vector stream, produced on the terminal device, into a sufficiently low bit-rate such that it can be sent across low bandwidth channels. This work proposes a compression technique based upon first transforming a block of feature vectors into a more compact matrix representation. Columns of the resulting matrix that correspond to faster temporal variation can be removed without loss in recognition performance. The number of bits allocated to the remaining coefficients in the matrix is determined automatically, based on a measure of the information present. Experiments show that the transform-based compression gives good recognition accuracy at bit rates of 4800, 2400 and 1200bps. For example at 1200bps the recognition performance is 98.03% compared to 98.57% with uncompressed speech.

This paper examines problems associated with performing speech recognition over mobile and IP networks. The main problems are identified as codec-based distortion and from speech vectors being lost from packet loss in the network. A realistic model for packet loss is developed, based on a three state Markov model and is shown to be capable of simulating the burst-like nature of packet loss. A two stage packet loss detection and estimation scheme is proposed and is shown to improve the recognition performance in the event of feature vectors being lost. Results from the Aurora database show that burst-like packet loss reduces the digit accuracy from 99% to 57% at 50% packet loss. Estimation of the lost packets recovers the performance to 77%.

This work looks at the issues involved in performing robust speech recognition over a packet-based network such as the IP network. This involves the combination of robust speech recognition together with a reliable method of sending speech data over the IP network. The format in which the speech is sent over the network is considered and results show that much better robustness is achieved when the front-end features are transmitted directly rather than encoding the speech with a codec. The problem of packet loss is addressed and a novel detection and estimation scheme for missing frames is introduced to overcome this problem. This is shown to recover performance with 50% packet loss from 33% to 90% which is only 3% below the no loss case.

This work looks at the issues involved in performing robust speech recognition over mobile and IP networks. The conventional method for sending speech across a mobile or IP network is to encode the speech on the terminal device using a low bit-rate codec and then transmit the stream of codec parameters. It is shown in this work that for speech recognition applications an alternative is available whereby the front-end processing part of a network-based speech recogniser is detached and moved onto the terminal device. Recognition features are then sent over the network to the remote recogniser. Simulations demonstrate that sending the speech features in this manner can provide a significant enhancement in recognition performance over the traditional codec-based approach. This technique forms the basis of the ETSI (European Telecommunications Standards Institute) Aurora standard. Problems arising with access over IP networks are also considered and in particular that of packet loss. A novel two-stage identification and estimation strategy is introduced which compensates for this loss of speech packets. Simulation results show that an almost negligible loss in recognition performance is possible at packet losses of up to 50%.

A combination of techniques for increasing recognition accuracy has been developed for an automated corporate directory system with 120,000 entries. Using a traditional recogniser an accuracy of around 60% has previously been obtained for both a 156 town name task and 1108 road name task. Techniques presented in this paper comprise front-end modifications, context dependent models, improved lexicon and noise modelling. This resulted in an increased recognition accuracy of around 90%.

This paper explores the modelling of phonetic segments of speech with multi-resolution spectral/time correlates. For spectral representation a set of multi-resolution cepstral features are proposed. Cepstral features obtained from a DCT of the log energy-spectrum over the full voice-bandwidth (100-4000 Hz) are combined with higher resolution features obtained from the DCT of upper subband (say 100-2100) and lower subband (2100-4000) halves. This approach can be extended to several levels of different resolutions. For representation of the temporal structure of speech segments or phonetic units, the conventional cepstral and dynamic cepstral features representing speech at the sub-phonetic levels, are supplemented by a set of phonetic features that describe the trajectory of speech over the duration of a phonetic unit. A conditional probability model for phonetic and sub-phonetic features is considered. Experiments demonstrate that the inclusion of the segmental features result in about 10% decrease in error rates.

Several noise compensation schemes for speech recognition in impulsive and nonimpulsive noise are considered. The noise compensation schemes are spectral subtraction, HMM-based Wiener (1949) filters, noise-adaptive HMMs, and a front-end impulsive noise removal. The use of the cepstral-time matrix as an improved speech feature set is explored, and the noise compensation methods are extended for use with cepstral-time features. Experimental evaluations, on a spoken digit database, in the presence of ear noise, helicopter noise, and impulsive noise, demonstrate that the noise compensation methods achieve substantial improvement in recognition across a wide range of signal-to-noise ratios. The results also show that the cepstral-time matrix is more robust than a vector of identical size, which is composed of a combination of cepstral and differential cepstral features.

This paper presents experimental results on the use of noise compensation schemes with hidden Markov model (HMM) speech recognition systems operating in the presence of impulsive noise. A measure of signal to impulsive noise ratio is introduced, and the effects of varying the percentage of impulsive noise contamination, and the power of impulsive noise, on speech recognition are investigated. For the modelling of an impulsive noise process, an amplitude-modulated binary sequence model and a binary-state HMM are considered. For impulsive noise compensation a front-end method and a noise-adaptive method are evaluated. Experiments demonstrate that the noise compensation methods achieve a substantial improvement in speech recognition accuracy across a wide range of signal to impulsive noise ratios.

This paper presents an analysis of the cepstral-time matrix. The coefficients of the cepstral-time matrix are found to be similar to the standard cepstral vector with differential features augmented on. It is also shown that the cepstral-time matrix is inherently robust to convolutional channel distortion. Spectral-subtraction, Wiener filtering and model combination are extended into two-dimensions where improved noise robustness is achieved. Experimental results using the NOISEX database with noise and channel distorted speech are presented.,

Conventional HMMs assume that speech spectral vectors are uncorrelated. The use of information on the temporal evolution of spectral features, within each state, can improve recognition accuracy and produce a more robust recognition system. The authors present experimental results on improvements in speech recognition using cepstral-time matrix units. Experimental evaluation using a spoken digit data base and a spoken alphabet data base, indicates that the use of cepstral-time matrix features in noisy conditions can provide an improvement in recognition of as much as 20% in comparison to a conventional spectral vector comprising of cepstral, delta cepstral and delta-delta cepstral features.>

This paper explores the advantages of using cepstral-time feature matrices, and spectral-time filters, for noisy speech recognition within a hidden Markov model framework. The use of cepstral-time features with spectral subtraction and state-based time-varying Wiener filters is investigated. Experimental results indicate that cepstral-time features, and spectral-time noise processing, provide an effective framework for robust speech recognition in noisy environments.>