© 2018 International Speech Communication Association. All rights reserved. In speech pathology, new assistive technologies using ASR and machine learning approaches are being developed for detecting speech disorder events. Classically-trained ASR model tends to remove disfluencies from spoken utterances, due to its focus on producing clean and readable text output. However, diagnostic systems need to be able to track speech disfluencies, such as stuttering events, in order to determine the severity level of stuttering. To achieve this, ASR systems must be adapted to recognise full verbatim utterances, including pseudo-words and non-meaningful part-words. This work proposes a training regime to address this problem, and preserve a full verbatim output of stuttering speech. We use a lightly-supervised approach using task-oriented lattices to recognise the stuttering speech of children performing a standard reading task. This approach improved the WER by 27.8% relative to a baseline that uses word-lattices generated from the original prompt. The improved results preserved 63% of stuttering events (including sound, word, part-word and phrase repetition, and revision). This work also proposes a separate correction layer on top of the ASR that detects prolongation events (which are poorly recog-nised by the ASR). This increases the percentage of preserved stuttering events to 70%.

Dysarthria is a neurological speech impairment, which usually results in the loss of motor speech control due to muscular atrophy and incoordination of the articulators. As a result the speech becomes less intelligible and difficult to model by machine learning algorithms due to inconsistencies in the acoustic signal and data sparseness. This paper presents phase-based feature representations for dysarthric speech that are exploited in the group delay spectrum. Such representations are found to be better suited to characterising the resonances of the vocal tract, exhibit better phone discrimination capabilities in dysarthric signals and consequently improve ASR performance. All the experiments were conducted using the UASPEECH corpus and significant ASR gains are reported using phase-based cepstral features in comparison to the standard MFCCs irrespective of the severity of the condition.

Copyright © 2017 ISCA. To help people who have lost their voice following total laryngectomy, we present a speech restoration system that produces audible speech from articulator movement. The speech articulators are monitored by sensing changes in magnetic field caused by movements of small magnets attached to the lips and tongue. Then, articulator movement is mapped to a sequence of speech parameter vectors using a transformation learned from simultaneous recordings of speech and articulatory data. In this work, this transformation is performed using a type of recurrent neural network (RNN) with fixed latency, which is suitable for realtime processing. The system is evaluated on a phoneticallyrich database with simultaneous recordings of speech and articulatory data made by non-impaired subjects. Experimental results show that our RNN-based mapping obtains more accurate speech reconstructions (evaluated using objective quality metrics and a listening test) than articulatory-to-acoustic mappings using Gaussian mixture models (GMMs) or deep neural networks (DNNs). Moreover, our fixed-latency RNN architecture provides comparable performance to an utterance-level batch mapping using bidirectional RNNs (BiRNNs).

This paper describes a technique that generates speech acoustics from articulator movements. Our motivation is to help people who can no longer speak following laryngectomy, a procedure that is carried out tens of thousands of times per year in the Western world. Our method for sensing articulator movement, permanent magnetic articulography, relies on small, unobtrusive magnets attached to the lips and tongue. Changes in magnetic field caused by magnet movements are sensed and form the input to a process that is trained to estimate speech acoustics. In the experiments reported here this “Direct Synthesis” technique is developed for normal speakers, with glued-on magnets, allowing us to train with parallel sensor and acoustic data. We describe three machine learning techniques for this task, based on Gaussian mixture models, deep neural networks, and recurrent neural networks (RNNs). We evaluate our techniques with objective acoustic distortion measures and subjective listening tests over spoken sentences read from novels (the CMU Arctic corpus). Our results show that the best performing technique is a bidirectional RNN (BiRNN), which employs both past and future contexts to predict the acoustics from the sensor data. BiRNNs are not suitable for synthesis in real time but fixed-lag RNNs give similar results and, because they only look a little way into the future, overcome this problem. Listening tests show that the speech produced by this method has a natural quality that preserves the identity of the speaker. Furthermore, we obtain up to 92% intelligibility on the challenging CMU Arctic material. To our knowledge, these are the best results obtained for a silent-speech system without a restricted vocabulary and with an unobtrusive device that delivers audio in close to real time. This work promises to lead to a technology that truly will give people whose larynx has been removed their voices back.

Users with disabilities can greatly benefit from personalised voice-enabled environmental-control interfaces, but for users with speech impairments (e.g. dysarthria) poor ASR performance poses a challenge to successful dialogue. Statistical dialogue management has shown resilience against high ASR error rates, hence making it useful to improve the performance of these interfaces. However, little research was devoted to dialogue management personalisation to specific users so far. Recently, data driven discriminative models have been shown to yield the best performance in dialogue state tracking (the inference of the user goal from the dialogue history). However, due to the unique characteristics of each speaker, training a system for a new user when user specific data is not available can be challenging due to the mismatch between training and working conditions. This work investigates two methods to improve the performance with new speakers of a LSTM-based personalised state tracker: The use of speaker specific acoustic and ASRrelated features; and dropout regularisation. It is shown that in an environmental control system for dysarthric speakers, the combination of both techniques yields improvements of 3.5% absolute in state tracking accuracy. Further analysis explores the effect of using different amounts of speaker specific data to train the tracking system.

International audience

The generalisation of dialogue state tracking to unseen dialogue states can be very challenging.In a slot-based dialogue system, dialogue states lie in discrete space where distances between states cannot be computed.Therefore, the model parameters to track states unseen in the training data can only be estimated from more general statistics, under the assumption that every dialogue state will have the same underlying state tracking behaviour.However, this assumption is not valid.For example, two values, whose associated concepts have different ASR accuracy, may have different state tracking performance.Therefore, if the ASR performance of the concepts related to each value can be estimated, such estimates can be used as general features.The features will help to relate unseen dialogue states to states seen in the training data with similar ASR performance.Furthermore, if two phonetically similar concepts have similar ASR performance, the features extracted from the phonetic structure of the concepts can be used to improve generalisation.In this paper, ASR and phonetic structurerelated features are used to improve the dialogue state tracking generalisation to unseen states of an environmental control system developed for dysarthric speakers.

Model-free reinforcement learning has been shown to be a promising data driven approach for automatic dialogue policy optimization, but a relatively large amount of dialogue interactions is needed before the system reaches reasonable performance.Recently, Gaussian process based reinforcement learning methods have been shown to reduce the number of dialogues needed to reach optimal performance, and pre-training the policy with data gathered from different dialogue systems has further reduced this amount.Following this idea, a dialogue system designed for a single speaker can be initialised with data from other speakers, but if the dynamics of the speakers are very different the model will have a poor performance.When data gathered from different speakers is available, selecting the data from the most similar ones might improve the performance.We propose a method which automatically selects the data to transfer by defining a similarity measure between speakers, and uses this measure to weight the influence of the data from each speaker in the policy model.The methods are tested by simulating users with different severities of dysarthria interacting with a voice enabled environmental control system.

Spoken control interfaces are very attractive to people with severe physical disabilities who often also have a type of speech disorder known as dysarthria. This condition is known to decrease the accuracy of automatic speech recognisers (ASRs) especially for users with moderate to severe dysathria. In this paper we investigate how applying probabilistic dialogue management (DM) techniques can improve interaction performance of an environmental control system for such users. The effect of having access to different amounts of adaptation data, as well as using different vocabulary size for speakers of different intelligibilities is investigated. We explore the effect of adapting the DM models as the ASR performance increases, such as is the case in systems where more adaptation data is collected through system use. Improvements compared to a non-probabilistic DM baseline are seen both in terms of dialogue length and success rate, 9% and 25% mean relative improvement respectively. Looking at just the more severe dysarthric speakers these numbers rise 25% and 75% mean relative improvement. These improvements are higher when the ASR data adaptation amount is small. Further results show that a DM trained on data from multiple speakers outperform a DM trained on data from a single speaker.

This paper investigates the potential of a silent speech interface (SSI) based on Permanent Magnetic Articulography (PMA) to be used in applications involving unconstrained, phonetically rich speech. In previous work the SSI was evaluated on isolatedword and connected-digits recognition tasks with promising results. Furthermore, it was shown that PMA data is enough to distinguish between minimal pairs of consonants with the same manner and place of articulation but different voicing. The study presented in this paper extends previous work by investigating to what extent information about the speech production process can be recovered from PMA data. In particular, the three main aspects of speech production are investigated here: voicing, place of articulation and manner of articulation. The results show that PMA achieves comparable accuracy to using the audio signal for discriminating the place of articulation of consonants, but it provides little information regarding the voicing and manner of articulation.

The automatic recognition of disordered speech is a domain that is characterised by limited amounts of training data for each speaker and large intra- and inter-speaker variations. This paper is concerned with how best to train an acoustic models in these circumstances; in particular, we look at how to select data for a background model from a pool of speakers for a given target speaker. We show that rather than including data from all available speakers (the standard approach in the typical speech domain), significantly better accuracy can be achieved by carefully selecting which speakers should contribute. Different methods based on measuring acoustic closeness between speakers and ranking them accordingly are investigated, and on the UASpeech isolated word recognition task, we achieve a 11.5% relative improvement compared to the baseline which uses data from all speakers. Accuracies for speakers with moderate to severe impairments are shown to improve the most with one speaker classed as having `low' intelligibility gaining a 60% relative improvement in accuracy.

Recently there has been increasing interest in ways of using out-of-domain (OOD) data to improve automatic speech recognition performance in domains where only limited data is available. This paper focuses on one such domain, namely that of disordered speech for which only very small databases exist, but where normal speech can be considered OOD. Standard approaches for handling small data domains use adaptation from OOD models into the target domain, but here we investigate an alternative approach with its focus on the feature extraction stage: OOD data is used to train feature-generating deep belief neural networks. Using AMI meeting and TED talk datasets, we investigate various tandem-based speaker independent systems as well as maximum a posteriori adapted speaker dependent systems. Results on the UAspeech isolated word task of disordered speech are very promising with our overall best system (using a combination of AMI and TED data) giving a correctness of 62.5 an increase of 15% on previously best published results based on conventional model adaptation. We show that the relative benefit of using OOD data varies considerably from speaker to speaker and is only loosely correlated with the severity of a speaker's impairments.

One of the main clinical applications of speech technology is in voice-enabled assistive technology for people with disordered speech. Progress in this area is hampered by a sparseness in suitable data and recent research have focused on ways of incorporating knowledge about typical (i.e., un-impaired) speech through the use of e.g., deep belief neural networks. This paper presents a new way of using deep belief neural networks trained on typical speech, namely to improve pronunciations for individual speakers. Analysis of the posterior probabilities show a clear correlation between measured pronunciation ‘disorderedness’ and the overall speech recognition performance of the full system. Based on this, we propose a method to use deep belief network outputs to i) identify which words are pronounced differently than what would be expected from a typical pronunciation, and ii) subsequently generate new pronunciations. We investigate different methods for pronunciation generation as well as what is the best way of using the modified pronunciations to inform the system development stages. Using the UAspeech database of disordered speech, we demonstrate improvement in average accuracy of 69.76% to 70.51%, with some speakers showing individual improvements of up to 10%.

Speech technologies are more important every day to assist people with speech disorders. They can help to increase their quality of life or help clinicians to make a diagnosis. In this paper a new methodology based on a total variability subspace modelled by factor analysis is proposed to assess the intelligibility of people with dysarthria. The acoustic information of each recording is efficiently compressed and a Pearson correlation of 0.91 between the vectors in this subspace (iVectors) and the intelligibility is obtained. As acoustic information only perceptual linear prediction features are used. The experiments are conducted on Universal Access Speech database. Also a new error metric to overcome the subjectivity in the intelligibility labels is proposed.

Speech-driven assistive technology can be an attractive alternative to conventional interfaces for people with physical disabilities. However, often the lack of motor-control of the speech articulators results in disordered speech, as condition known as dysarthria. Dysarthric speakers can generally not obtain satisfactory performances with off-the-shelf automatic speech recognition (ASR) products and disordered speech ASR is an increasingly active research area. Sparseness of suitable data is a big challenge. The experiments described here use UAspeech, one of the largest dysarthric databases available, which is still easily an order of magnitude smaller than typical speech databases. This study investigates how far fundamental training and adaptation techniques developed in the LVCSR community can take us. A variety of ASR systems using maximum likelihood and MAP adaptation strategies are established with all speakers obtaining significant improvements compared to the baseline system regardless of the severity of their condition. The best systems show on average 34% relative improvement on known published results. An analysis of the correlation between intelligibility of the speaker and the type of system which would represent an optimal operating point in terms of performance shows that for severely dysarthric speakers, the exact choice of system configuration is more critical than for speakers with less disordered speech.

This paper presents a novel noise-robust automatic speech recognition (ASR) system that combines aspects of the noise modeling and source separation approaches to the problem. The combined approach has been motivated by the observation that the noise backgrounds encountered in everyday listening situations can be roughly characterized as a slowly varying noise floor in which there are embedded a mixture of energetic but unpredictable acoustic events. Our solution combines two complementary techniques. First, an adaptive noise floor model estimates the degree to which high-energy acoustic events are masked by the noise floor (represented by a soft missing data mask). Second, a fragment decoding system attempts to interpret the high-energy regions that are not accounted for by the noise floor model. This component uses models of the target speech to decide whether fragments should be included in the target speech stream or not. Our experiments on the CHiME corpus task show that the combined approach performs significantly better than systems using either the noise model or fragment decoding approach alone, and substantially outperforms multicondition training.

In previous publications, a silent speech interface based on permanent-magnetic articulography (PMA) has been introduced and evaluated using standard automatic speech recognition techniques. However, word recognition is a task that is computationally expensive and introduces a significant time delay between speech articulation and generation of the acoustic signal. This paper investigates a direct synthesis approach where control parameters for parametric speech synthesis are generated directly from the sensor data of the silent speech interface, without an intermediate lexical representation. Users of such a device would not be tied to the limited vocabulary of a word-based recogniser and could therefore express themselves more freely. This paper presents a feasibility study that investigates whether it is possible to infer speech synthesis parameters from PMA sensor data

This paper addresses the problem of speech recognition using distant binaural microphones in reverberant multisource noise conditions. Our scheme employs a two stage fragment decoding approach: first spectro-temporal acoustic source fragments are identified using signal level cues, and second, a hypothesisdriven stage simultaneously searches for the most probable speech/background fragment labelling and the corresponding acoustic model state sequence. The paper reports the first successful attempt to use binaural localisation cues within this framework. By integrating binaural cues and acoustic models in a consistent probabilistic framework, the decoder is able to derive significant recognition performance benefits from fragment location estimates despite their inherent unreliability.

We present a new corpus designed for noise-robust speech processing research, CHiME. Our goal was to produce material which is both natural (derived from reverberant domestic environments with many simultaneous and unpredictable sound sources) and controlled (providing an enumerated range of SNRs spanning 20 dB). The corpus includes around 40 hours of background recordings from a head and torso simulator positioned in a domestic setting, and a comprehensive set of binaural impulse responses collected in the same environment. These have been used to add target utterances from the Grid speech recognition corpus into the CHiME domestic setting. Data has been mixed in a manner that produces a controlled and yet natural range of SNRs over which speech separation, enhancement and recognition algorithms can be evaluated. The paper motivates the design of the corpus, and describes the collection and post-processing of the data. We also present a set of baseline recognition results.

This paper reports on isolated word recognition experiments using a novel silent speech interface. The interface consist of magnetic pellets that are fixed to relevant speech articulators, and a set of magnetic field sensors that measure changes in the overall magnetic field created by these pellets during speech. The reported experiments demonstrate the effectiveness of this technique and show the suitability of the system, even at its early stages of development, for small vocabulary speech recognition.

Speech has a property that the speech unit preceding a speech pause tends to lengthen. This work presents the use of a dynamic Bayesian network to model the prepausal lengthening effect for robust speech recognition. Specifically, we introduce two distributions to model inter-state transitions in prepausal and non-prepausal words, respectively. The selection of the transition distributions depends on a random variable whose value is influenced by whether a pause will appear between the current and the following word. Two experiments are presented here. The first one considers pauses hypothesised during speech decoding. The second one employs an extra component for speech/non-speech determination. By modelling the prepausal lengthening effect we achieve a 5.5% relative reduction in word error rate on the 500-word task of the SVitchboard corpus.

Abstract Inbuildingpersonalized syntheticvoicesforpeople withspeechdisorders, the output should capture the individual’s vocal iden-tity. This paper reports a listener judgment experiment on thesimilarity of Hidden Markov Model based synthetic voices us-ing varying amounts of adaptation data to two non-impairedspeakers. We conclude that around 100 sentences of data isneeded to build a voice that retains the characteristics of thetarget speaker but using more data improves the voice. Experi-ments using Multi-Layer Perceptrons (MLPs) are conducted tofind which acoustic features contribute to the similarity judg-ments. Results show that mel-cepstral distortion and fraction ofvoicing agreement contribute most to replicating the similarityjudgment but the combination of all features is required for ac-curate prediction. Ongoing work applies the findings to voicebuilding for people with impaired speech. Index Terms : HMM-based speech synthesis, voice similaritymeasure, voice banking 1. Introduction

This study reports on the development of a diagnostic expert system – incorporating a multilayer perceptron (MLP) – designed to identify any sub-type of dysarthria (loss of neuromuscular control over the articulators) manifested by a patient undergoing a Frenchay Dysarthria Assessment (FDA) evaluation. If sufficient information is provided describing pathological features of the patient’s speech, the rule-based classifier (RBC) can out-perform the MLP in terms of rendering a more accurate and consistent diagnosis. The combination MLP/RBC developed during this study realised an overall improvement in diagnostic accuracy of 9.3% (absolute) for a selection of dysarthric cases, representing a substantial improvement over the benchmark system which – unlike the MLP/RBC – cannot directly process acoustic data.

When speech is corrupted by other sound sources certain spectro-temporal regions will be dominated by speech energy and others by the noise. Listeners are able to exploit these cues to achieve robust speech perception in adverse conditions. Inspired by this perception process a ‘speech fragment decoding’ technique has shown promising robustness when handling multiple sound sources. This paper proposes an approach to estimating ‘speechiness’ – a degree of confidence that a spectrotemporal region is dominated by speech energy – using the modulation spectrogram. This additional knowledge is employed to steer the decoder towards selecting more reliable speech evidence in noise. Experiments show that the speechiness measure is capable of improving recognition accuracies in various noise conditions at 0 dB global signal-to-noise ratio.

Conventional HMMs have weak duration constraints. In noisy conditions, the mismatch between corrupted speech signals and models trained on clean speech may cause the decoder to produce word matches with unrealistic durations. This paper presents a simple way to incorporate word duration constraints by unrolling HMMs to form a lattice where word duration probabilities can be applied directly to state transitions. The expanded HMMs are compatible with conventional Viterbi decoding. Experiments on connected-digit recognition show that when using explicit duration constraints the decoder generates word matches with more reasonable durations, and word error rates are significantly reduced across a broad range of noise conditions.

Autocorrelograms exhibit tree-like structures whose spines are located at a delay of 1/F0. This paper exploits the dendritic autocorrelogram structure for the identification of spectro-temporal regions dominated by a single periodic sound source in monaural acoustic mixtures. Each frame of the mixture is first segmented into different sound sources in the autocorrelogram domain. Local pitch estimates are formed for each source and used as a cue for temporal integration. A confidence score is computed for each time-frequency pixel in the grouped regions to determine its probability of belonging to the group. The system is evaluated using simultaneous speech in a coherence measuring experiment and also employed within an ASR system where it produces improved results for the Interspeech 2006 Speech Separation Challenge. Index Terms: speech separation, correlogram, multipitch tracking.

Abstract Speech technology is potentially of enormous benefit to people with physical disabilities. Applications of speech technology to e-inclusion are reviewed and described in the areas of access, control, communication and rehabilitation/therapy, with particular reference to speech technology developments for people with disordered speech. To be successful, applications should effectively take into account the needs of user groups and have the ability to adapt to the needs of individuals. This is a challenging area but effective progress can be made through multi-disciplinary research and development. 1. Introduction People with physical disabilities take advantage of a variety of methods to gain access to information technology and to electronic assistive technology for communication, mobility and daily living tasks. Many of these access methods are slow and can lead to frustration, a prime example being the use of switch-activated menu scanning (hereafter referred to as switch-scanning). Automatic Speech Recognition is potentially of enormous benefit to people with severe physical disabilities. The tremendous richness of human speech communication gives the user many degrees of freedom for control and input. The speed of speech recognition also gives it a potential advantage over other input methods commonly employed by physically disabled people. People with neurological conditions causing disability often have associated dysarthria, which is the most common acquired speech disorder affecting 170 per 100,000 population [1]. This may be developmental dysarthria such as that associated with cerebral palsy or acquired dysarthria associated with progressive neurological disease (e.g. Parkinson's disease, motor neurone disease, and multiple sclerosis) or sudden onset conditions such as/or head injury. In its severest form, dysarthric speech is unintelligible to others and may take the form of producing vocal utterances, rather than words recognisable to unfamiliar communication partners. The combination of speech and general physical disability can make it particularly problematic for people to interact in their environment and can severely limit independence and inclusion. This paper examines some of the areas for e-inclusion of people with disabilities that can benefit from the use of speech technology. These areas include access, control, communication, rehabilitation and therapy. In all areas, the use of speech technology for people with dysarthric speech is particularly examined and discussed.

Conventional hidden Markov models (HMMs) have weak duration constraints. This may cause the decoder to produce word matches with unrealistic durations in noisy situations. This paper describes techniques for modelling context-dependent word duration cues and incorporating them directly in a multi-stack decoding algorithm. The proposed model is capable of penalising duration constraints of a word depending on its context. Experiments on connected digit recognition show that the new system can significantly improve recognition performance at different noise levels. 1.

This paper presents an application of missing data techniques in speech enhancement. The enhancement system consists of two stages: the first stage uses a recurrent neural network, which is supplied with noisy speech and produces enhanced speech; whereas the second stage uses missing data techniques to further improve the quality of enhanced speech. The results suggest that combining missing data technique with RNN enhancement is an effective enhancement scheme resulting in a 16 dB background noise reduction for all input signal to noise ratio (SNR) conditions from -5 to 20 dB, improved spectral quality and robust automatic speech recognition performance.

This study reports on the development of an automated isolated-word intelligibility metric system designed to improve the scoring consistency and reliability of the Frenchay Dysarthria Assessment Test (FDA).The proposed intelligibility measurements are based on the probabilistic likelihood scores derived from the forced alignment of the dysarthric speech to whole-word hidden Markov models (HMMs) trained on data from a variety of normal speakers. The hypothesis is that these probability scores are correlated to the decoding effort made by naive listeners when trying to comprehend dysarthric utterances. Initial results indicate that the scores returned from these composite measurements provide a more fine-grained assessment of a given dysarthric individual’s oral communicative competence when compared with traditional ‘right-or-wrong’ scoring of expert listeners’ transcriptions of dysarthric speech samples.

The word error rate (WER), commonly used in ASR assessment, measures the cost of restoring the output word sequence to the original input sequence. However, for most CSR applications apart from dictation machines a more meaningful performance measure would be given by the proportion of information communicated. In this article we introduce two new absolute CSR performance measures: MER (match error rate) and WIL (word information lost). MER is the proportion of I/O word matches which are errors. WIL is a simple approximation to the proportion of word information lost which overcomes the problems associated with the RIL (relative information lost) measure that was proposed half a century ago. Issues relating to ideal performance measurement are discussed and the commonly used Viterbi input/output alignment procedure, with zero weight for hits and equal weight for substitutions, deletions and insertions, is shown to be optimal.

We describe an unusual ASR application: recognition of command words from severely dysarthric speakers, who have poor control of their articulators. The goal is to allow these clients to control assistive technology by voice. While this is a small vocabulary, speaker-dependent, isolated-word application, the speech material is more variable than normal, and only a small amount of data is available for training. After training a CDHMM recogniser, it is necessary to predict its likely performance without using an independent test set,so that confusable words can be replaced by alternatives. We present a battery of measures of consistency and confusability, based on forced-alignment, which can be used to predict recogniser performance. We show how these measures perform, and how they are presented to the clinicians who are the users of the system. 1.

Computer based speech training systems aim to provide the client with customised tools for improving articulation based on audio-visual stimuli and feedback. They require the integration of various components of speech technology, such as speech recognition and transcription tools, and a database management system which supports multiple on-the-fly configurations of the speech training application. This paper describes the requirements and development of STRAPTk ( www.dcs.shef.ac.uk/spandh/projects/straptk-Speech Training Application Toolkit) from the point of view of developers, clinicians, and clients in the domain of speech training for severely dysarthric speakers. Preliminary results from an extended field trial are presented. 1.

In this study, techniques for classification with missing or unreliable data are applied to the problem of noise-robustness in Automatic Speech Recognition (ASR). The techniques described make minimal assumptions about any noise background and rely instead on what is known about clean speech. A system is evaluated using the Aurora 2 connected digit recognition task. Using models trained on clean speech we obtain a 65% relative improvement over the Aurora clean training baseline system, a performance comparable with the Aurora baseline for multicondition training.

In the missing data' approach to improving the robustness of automatic speech recognition to added noise, an initial process identifies spectral- temporal regions which are dominated by the speech source. The remaining regions are considered to bemissing'. In this paper we develop a connectionist approach to the problem of adapting speech recognition to the missing data case, using Recurrent Neural Networks. In contrast to methods based on Hidden Markov Models, RNNs allow us to make use of long-term time constraints and to make the problems of classification with incomplete data and imputing missing values interact. We report encouraging results on an isolated digit recognition task. 1. Introduction Automatic Speech Recognition systems perform reasonably well in controlled and matched training and recognition conditions. However, performance deteriorates when there is a mismatch between training and testing conditions, caused for instance by additive noise (Lippmann, 1997). Conventional techniques for improving recognition robustness (reviewed by Furui 1997) seek to eliminate or reduce the mismatch, for instance by enhancement of the noisy speech, by adapting statistical models for speech units to the noise condition or simply by training in different noise conditions. Missing data techniques provide an alternative solution for speech corrupted by additive noise which make minimal assumptions about the nature of the noise. They are based on identifying uncorrupted, reliable regions in the frequency domain and adapting recognition algorithms so that classification is based on these regions. Present missing data techniques developed at Sheffield (Barker et al. 2000a, Barker et al. 2000b, Cooke et al., 2001) and elsewhere (Drygaglo et al., 1998, Raj et al., 2000) adapt the prevailing technique for ASR based on Continuous Density Hidden Markov Models. CDHMMs are generative models which do not give direct estimates of posterior probabilities of the classes given the acoustics. Neural Networks, unlike HMMs, are discriminative models which do give direct estimates of posterior probabilities and have been used with success in hybrid ANN/HMM speech recognition systems (Bourlard et al., 1998). In this paper, we adapt a recurrent neural network architecture introduced by (Gingras & Bengio, 1998) for robust ASR with missing data. Missing data techniques for Robust ASR 2.1 Missing data masks Speech recognition with missing data is based on the assumption that some regions in time/frequency remain uncorrupted for speech with added noise. See (Cooke et al., 2001) for arguments to support this assumption. Initial processes, based on local signal-to-noise estimates, on auditory grouping cues, or a combination (Barker et al., 2001) define a binary missing data mask': ones in the mask indicate reliable (orpresent') features and zeros indicate unreliable (or `missing') features. 2.2 Classification with missing data Techniques for classification with incomplete data can be divided into imputation and marginalisation. Imputation is a technique in which missing features are replaced by estimated values to allow the recognition process proceed in normal way. If the missing values are replaced by either zeros, random values or their means based on training data, the approach is called unconditional imputation. On the other hand in conditional imputation conditional statistics are used to estimate the missing values given the present values. In the marginalisation approach missing values are ignored (by integrating over their possible ranges) and recognition is performed with the reduced data vector which is considered reliable. For the multivariate mixture Gaussian distributions used in CDHMMs, marginalisation and conditional imputation can be formulated analytically (Cooke et al., 2001). For missing data ASR further improvements in both techniques follow from using the knowledge that for spectral energy features the unreliable data is bounded between zero and the energy in speech+noise mixture (Vizinho et al., 1999), (Josifovski et al., 1999). These techniques are referred to as bounded marginalisation and bounded imputation. Coupled with a `softening' of the reliable/unreliable decision, missing data techniques produce good results on a standard connected-digits-in-noise recognition task: performance using models trained on clean data is comparable, and in severe noise superior, to conventional systems trained across different noise conditions (Barker et al., 2001). 2.3 Why recurrent neural nets for missing data robust ASR? Several neural net architectures have been proposed to deal with the missing data problem in general (Ahmed & Tresp, 1993), (Ghahramani & Jordan, 1994). The problem in using neural networks with missing data is to compute the output of a node/unit when some of its input values are unavailable. For marginalisation, this involves finding a way of integrating over the range of the missing values. A robust ASR system to deal with missing data using neural networks has recently been proposed by (Morris et al., 2000). This is basically a radial basis function neural network with the hidden units associated with a diagonal covariance gaussian. The marginal over the missing values can be computed in this case and hence the resulting system is equivalent to the HMM based missing data speech recognition system using marginalisation. Reported performance is also comparable to that of the HMM based speech recognition system. In this paper missing data is dealt with by imputation. We use recurrent neural networks to estimate missing values in the input vector. RNNs have the potential to capture long-term contextual effects over time, and hence to use temporal context to compensate for missing data which CDHMM based missing data techniques do not do. The only contextual information available in CDHMM decoding come from the addition of temporal derivatives to the feature vector. RNNs also allow a single net to perform both imputation and classification, with the potential of combining these processes to mutual benefit. The RNN architecture proposed by Gingras et al. (1998) is based on a fully-connected feedforward network with input, hidden and output layers using hyperbolic tangent activation functions. The output layer has one unit for each class and the network is trained with the correct classification as target. Recurrent links are added to the feedforward net with unit delay from output to the hidden units as in Jordan networks (Jordan, 1988). There are also recurrent links with unit delay from hidden units to missing input units to impute missing features. In addition, there are self delayed terms with a fixed weight for each unit which basically serve to stabilise RNN behaviour over time and help in imputation as well. Gingras et al. used this RNN both for a pattern classification task with static data (one input vector for each example) and sequential data (a sequence of input values for each example). Our aim is to adapt this architecture for robust ASR with missing data. Some preliminary static classification experiments were performed on vowel spectra (individual spectral slices excised from the TIMIT database). RNN performance on this task with missing data was better than standard MLP and gaussian classifiers. In the next section we show how the net can be adapted for dynamic classification of the spectral sequences constituting words. RNN architecture for robust ASR with missing data Figure 1 illustrates our modified version of the Gingras and Bengio architecture. Instead of taking feedback from the output to the hidden layer we have chosen a fully connected or Elman RNN (Elman, 1990) where there are full recurrent links from the past hidden layer to the present hidden layer (figure 1). We have observed that these links produce faster convergence, in agreement with (Pedersen, 1997). The number of input units depends on the size of feature vector, i.e. the number of spectral channels. The number of hidden units is determined by experimentation. There is one output unit for each pattern class. In our case the classes are taken to be whole words, so in the isolated digit recognition experiments we report, there are eleven output units, for 1' -9', zero' andoh'. In training, missing inputs are initialised with their unconditional means. The RNN is then allowed to impute missing values for the next frame through the recurrent links, after a feedforward pass. Where is the missing feature at time t, is the learning rate, indicates recurrent links from a hidden unit to the missing input and is the activation of hidden unit j at time t-1. The average of the RNN output over all the frames of an example is taken after these frames have gone through a forward pass. The sum squared error between the correct targets and the RNN output for each frame is back-propagated through time and RNN

In previous work we have developed the theory and demonstrated the promise of the Missing Data approach to robust Automatic Speech Recognition. This technique is based on hard decisions as to whether each time-frequency "pixel" is either reliable or unreliable. In this paper we replace these discrete decisions with soft estimates of the probability that each "pixel" is reliable. We adapt the probability calculation to use these estimates as weighting factors for the complementary reliable/unreliable interpretations for each feature vector component. Experiments using the TIDigits connected digit recognition task demonstrate that this technique affords significant performance improvements at low SNRs. 1. INTRODUCTION In previous work [2, 5, 6] we have developed the theory and demonstrated the promise of the Missing Data approach to robust Automatic Speech Recognition. In this technique, spectral-temporal regions uncontaminated by noise are identified and CDHMM recognition methods are ...

If the data vector for input to an automatic classifier is incomplete, the optimal estimate for each class probability must be calculated as the expected value of the classifier output. We identify a form of RBF classifier whose expected outputs can easily be evaluated in terms of the original function parameters. We then describe two ways in which this classifier can be applied to robust automatic speech recognition, depending on whether or not the position of missing data is known

In this paper we are presenting a new algorithm for determining the fundamental frequency.The evaluation of pitch is a very difficult problem mainly because of the great variability and irregularity of the speech signals.The algorithm we are presenting is original so far as it relies on the implicit calculation of the autocorrelation of the temporal excitation signal.We have tested our algorithm on the Bagshaw database, created at the Center for Speech Technology Research at Edinburgh, which is primarily dedicated to the evaluation of algorithms estimating the fundamental frequency of speech.The results of our experiments show that our approach is very reliable.

We address the theoretical and practical issues involved in automatic speech recognition (ASR) when some of the observation data for the target signal is masked by other signals. Techniques discussed range from simple missing data imputation to Bayesian optimal classification. We have developed the Bayesian approach because this allows prior knowledge to be incorporated naturally into the recognition process, thereby permitting us to go beyond the simple "integrate over missing data" or "marginals" approach reported elsewhere, which we show to be inadequate for dealing with realistic patterns of missing data. After deriving general techniques for recognition with missing data, these techniques are formulated in the context of an HMM based CSR system. This scheme is evaluated under both random and more realistic patterns of missing data, with speech from the DARPA RM corpus and noise from NOISEX. We find that a key problem in real world recognition with missing data is that efficient ASR requires data vector components to be independent, and incomplete data cannot be orthogonalised in the usual way by projection. We show that use of spectral peaks only can provide an effective solution to this problem.

In noisy listening conditions, the information available on which to base speech recognition decisions is necessarily incomplete: some spectro-temporal regions are dominated by other sources. We report on the application of a variety of techniques for missing data in speech recognition. These techniques may be based on marginal distributions or on reconstruction of missing parts of the spectrum. Application of these ideas in the resource management task shows a performance which is robust to random removal of up to 80% of the frequency channels, but falls off rapidly with deletions which more realistically simulate masked speech. We report on a vowel classification experiment designed to isolate some of the RM problems for more detailed exploration. The results of this experiment confirm the general superiority of marginals-based schemes, demonstrate the viability of shared covariance statistics, and suggest several ways in which performance improvements on the larger task may be obtained.

Traditional speech training methods can prove cumbersome because of the difficulty of providing the subject with good feedback, maintaining her/his motivation over long periods and stabilising the improvement in articulation. In this work we provide visual feedback based on displaying a trajectory in a 2Dimensional „phonetic space‟. Data are presented from a small-scale efficacy study, which illustrate the use of OLT in speech therapy for misarticulated sibilant fricatives. Results for the contrasting articulations are compared and the potential of OLT as a therapeutic technique is discussed.

We describe a novel paradigm for automatic speech recognition in noisy environments in which an initial stage of auditory scene analysis separates out the evidence for the speech to be recognised from the evidence for other sounds. In general, this evidence will be incomplete, since intruding sound sources will dominate some spectro-temporal regions. We generalise continuous-density hidden Markov model recognition to this 'occluded speech' case. The technique is based on estimating the probability that a Gaussian mixture density distribution for an auditory firing rate map will generate an observation such that the separated components are at their observed values and the remaining components are not greater than their values in the acoustic mixture. Experiments on isolated digit recognition in noise demonstrate the potential of the new approach to yield performance comparable to that of listeners.

In this paper, we propose a new paradigm for robust ASR based on auditory scene analysis. In previous work, we have shown how models of auditory processing and grouping principles can be used to separate the evidence for a speech signal from arbitrary intrusions. However, this evidence will generally be incomplete since some spectrotemporal regions will be dominated by the other sources. Here, we address the problem of recognising such `occluded' speech. Two investigations are reported: the first applies unsupervised learning and subsequent recognition to spectral vectors with missing components. The second adapts the Viterbi algorithm for HMM-based ASR to the occluded speech case. Both techniques are encouragingly robust: for instance, more than half of the observation vector can be obscured without appreciable deterioration in recognition performance. Additionally, our demonstration that it is possible to learn to recognise speech from partial information suggests a model for the for...

We propose an approach to acoustic-phonetic processing in speech recognition systems based on an intermediate representation, the 'Speech Sketch', whose purpose is to bridge the representational gap between acoustic evidence and phonetic knowledge. The Speech Sketch makes parameter behaviour, rather than parameter values, explicit. We report on a small scale testbed for this approach. In this work we have attempted to discriminate between the semivowels in vowel-semivowel-vowel utterances using, as our acoustic evidence, LPC-derived formant frequency estimates. Our Speech Sketch is based on linear descriptions and our acoustic-phonetic knowledge is represented in frame-like structures. For three male speakers the performance of this software is comparable with that of a human expert presented with the same parametric data.