This work deals with non-native children's speech and investigates both multi-task and transfer learning approaches to adapt a multi-language Deep Neural Network (DNN) to speakers, specifically children, learning a foreign language. The application scenario is characterized by young students learning English and German and reading sentences in these second-languages, as well as in their mother language. The paper analyzes and discusses techniques for training effective DNN-based acoustic models starting from children's native speech and performing adaptation with limited non-native audio material. A multi -lingual model is adopted as baseline, where a common phonetic lexicon, defined in terms of the units of the International Phonetic Alphabet (IPA), is shared across the three languages at hand (Italian, German and English); DNN adaptation methods based on transfer learning are evaluated on significant non-native evaluation sets. Results show that the resulting non-native models allow a significant improvement with respect to a mono-lingual system adapted to speakers of the target language.

Abstract This paper introduces deep neural network (DNN)–hidden Markov model (HMM)-based methods to tackle speech recognition in heterogeneous groups of speakers including children. We target three speaker groups consisting of children, adult males and adult females. Two different kind of approaches are introduced here: approaches based on DNN adaptation and approaches relying on vocal-tract length normalisation (VTLN). First, the recent approach that consists in adapting a general DNN to domain/language specific data is extended to target age/gender groups in the context of DNN–HMM. Then, VTLN is investigated by training a DNN–HMM system by using either mel frequency cepstral coefficients normalised with standard VTLN or mel frequency cepstral coefficients derived acoustic features combined with the posterior probabilities of the VTLN warping factors. In this later, novel, approach the posterior probabilities of the warping factors are obtained with a separate DNN and the decoding can be operated in a single pass when the VTLN approach requires two decoding passes. Finally, the different approaches presented here are combined to take advantage of their complementarity. The combination of several approaches is shown to improve the baseline phone error rate performance by thirty per cent to thirty-five per cent relative and the baseline word error rate performance by about ten per cent relative.

This paper describes an approach for adapting a DNN trained on adult speech to children voices. The method extends a previous one, based on the Kullback-Leibler divergence between the original (adult) DNN output distribution and the target one, by accounting for the quality of the supervision of the adaptation utterances. In addition, starting from the observation that by gradually removing from the adaptation set the sentences with higher WERs significant performance improvements can be achieved, we also investigate the usage of automatic selection of adaptation utterances. For determining transcription quality we investigate the use of confidence estimates of recognized hypotheses. We present experiments and related results achieved on an Italian data set of children's speech. We show that the proposed DNN adaptation approach allows to significantly reduce the WER on a given test set from 14.2% (corresponding to using the non adapted DNN, trained on adult speech) to 10.6%. It is worth mentioning that the latter result has been achieved without making use of any training data specific of children's speech.

This paper introduces approaches based on vocal tract length normalisation (VTLN) techniques for hybrid deep neural network (DNN) - hidden Markov model (HMM) automatic speech recognition when targeting children's and adults' speech. VTLN is investigated by training a DNN-HMM system by using first mel frequency cepstral coefficients (MFCCs) normalised with standard VTLN. Then, MFCCs derived acoustic features are combined with the VTLN warping factors to obtain an augmented set of features as input to a DNN. In this later, novel, approach the warping factors are obtained with a separate DNN and the decoding can be operated in a single pass when standard VTLN approach requires two decoding passes. Both VTLN-based approaches are shown to improve phone error rate performance, up to 20% relative improvement, compared to a baseline trained on a mixture of children's and adults' speech.

This paper presents the results of an experimental study conducted with the aim of comparing two methods for crowdsourcing speech transcription that incorporate two different quality control mechanisms (i.e. explicit versus implicit) and that are based on two different processes (i.e. parallel versus iterative). In the Gold Standard method the same speech segment is transcribed in parallel by multiple contributors whose reliability is checked with respect to some reference transcriptions provided by experts. On the other hand, in the Dual Pathway method two independent groups of contributors work on the same set of transcriptions refining them in an iterative way until they converge, and thus eliminating the need to have reference transcriptions and to check transcription quality in a separate phase. These two methods were tested on about half an hour of broadcast news speech and for two different European languages, namely German and Italian. Both methods obtained good results in terms of Word Error Rate (WER) and compare well with the word disagreement rate of experts on the same data.

This paper describes two different automatic transcription systems developed for judicial application domains for the Polish and Italian languages. The judicial domain requires to cope with several factors which are known to be critical for automatic speech recognition, such as: background noise, reverberation, spontaneous and accented speech, overlapped speech, cross channel effects, etc. The two automatic speech recognition (ASR) systems have been developed independently starting from out-of-domain data and, then, they have been adapted using a certain amount of in-domain audio and text data. The ASR performance have been measured on audio data acquired in the courtrooms of Naples and Wroclaw. The resulting word error rates are around 40%, for Italian, and around between 30% and 50% for Polish. This performance, similar to that reported for other comparable ASR tasks (e.g. meeting transcriptions with distant microphone), suggests that possible applications can address tasks such as indexing and/or information retrieval in multimedia documents recorded during judicial debates.

In this paper, phone-to-word transduction is first investigated by coupling a speech recognizer, generating for each speech segment a phone sequence or a phone confusion network, with the efficient decoder of confusion networks adopted by MOSES, a popular statistical machine translation toolkit. Then, system combination is investigated by combining the outputs of several conventional ASR systems with the output of a system embedding phone-to-word decoding through statistical machine translation. Experiments are carried out in the context of a large vocabulary speech recognition task consisting of transcription of speeches delivered in English during the European Parliament Plenary Sessions (EPPS). While only a marginal performance improvements is achieved in system combination experiments when the output of the phone-to-word transducer is included in the combination, partial results show a great potential for improvements.

This paper addresses speaker adaptive acoustic modeling, based on feature space maximum likelihood linear regression, in the context of on-line telephony applications. An adaptive acoustic modeling method, that we previously proved effective in off-line applications, is used to train acoustic models to be used in text-dependent and text-independent on-line adaptation. Experiments on telephony speech data indicate that feature space maximum a posteriori linear regression (fMAPLR) greatly helps to cope with sparse adaptation data when performing instantaneous and incremental adaptation with both baseline models and speaker adaptively trained models. The use of speaker adaptively trained models in conjunction with fMAPLR leads to the best recognition results in both instantaneous and incremental adaptation. The proposed text-independent adaptation approach, exploiting speaker adaptively trained models, is also proven effective.

Current research has shown that the speech intelligibility in children with cleft lip and palate (CLP) can be estimated automatically using speech recognition methods. On German CLP data high and significant correlations between human ratings and the recognition accuracy of a speech recognition system were already reported. In this paper we investigate whether the approach is also suitable for other languages. Therefore, we compare the correlations obtained on German data with the correlations on Italian data. A high and significant correlation (r=0.76; p 0.05). Index Terms: speech recognition, speech intelligibility, cleft lip and palate.

Most state-of-the-art automatic transcription systems generate word transcriptions of the incoming audio data through two or more decoding passes interleaved by adaptation of acoustic models. It was proved that better results are obtained when the adaptation procedure exploits a supervision generated by a system different than the one under adaptation. In this paper, cross-system adaptation is investigated by using supervisions generated by several systems built varying the phoneme set and the acoustic front-end. Furthermore, an adaptation procedure is presented that makes use of multiple supervisions of the audio data for adapting the acoustic models within the MLLR framework. The gain achieved with cross-system adaptation and by adapting the acoustic models exploiting multiple, intra-site and cross-site, supervisions is demonstrated on the English European parliamentary speeches task.

This paper presents several acoustic analyses on read speech, collected from 5 adults and 35 children aged 5 to 17 years, focusing on consonants and consonant-vowel transition. Characteristics of consonants such as duration, intra-speaker variability and, for stop consonants, voice onset time are analyzed and compared with results achieved on vowels. Strong and significant correlation with age is observed for both duration and intra-speaker variability. In fact, younger children show higher phone duration and larger spectral and temporal variability than older children and adults. Voice onset time, on the other hand, is less correlated with age. Analysis of consonant-vowel transition shows that the duration of the transition and the amount of spectral difference between consonant and vowel are clearly age-dependent. Younger children, in fact, show shorter transition duration and larger spectral difference between consonant and vowel in the consonant-vowel pair

Adaptive training aims at reducing the influence of speaker, channel and environment variability on the acoustic models. We describe an acoustic normalization approach to adaptive training. Phonetically irrelevant acoustic variability is reduced at the beginning of the training procedure w.r.t. a set of target models. The set of target models can be a set of HMMs or a Gaussian mixture model (GMM). CMLLR is applied to normalize the acoustic features. The normalized data contains less unwanted variability and is used to generate and train the recognition models. Employing a GMM as a target model leads to a text-independent procedure that can be embedded into the acoustic front-end. On a broadcast news transcription task we obtain relative reductions in WER of 7.8% in the first recognition pass over a conventionally trained system and of 3.4% in the second recognition pass over a SAT-trained system.

This paper describes the corpus of recordings of children's speech which was collected as part of the EU FP5 PF STAR project.The corpus contains more than 60 hours of speech, including read and imitated native-language speech in British English, German and Swedish, read and imitated non-nativelanguage English speech from German, Italian and Swedish children, and native-language spontaneous and emotional speech in English and German. 1 This work was conducted as part of EU FP5 PF STAR (Preparing Future Multisensorial Interaction Research)

This paper presents an initial effort in the area of nonnative children’s speech recognition by exploiting two children databases, one consisting of speech collected from native English children, the other one consisting of English sentences read by Italian learners of English in the same age range of the native speakers. First, a baseline speech recognizer for British English was trained on the corpus of native speech and applied to recognize native and non-native speech. Word error rates achieved for Italian children were 100%-600% higher than those achieved for native English children of the same age. By using a small amount of non-native speech from a group of Italian learners of English, acoustic models were adapted to this particular category of speakers. Adaptation of both contextindependent and context-dependent HMMs showed to greatly improve recognition performance on non-native speech, obtaining relative reductions in word error rate up to 66.2%.

In this paper, a novel speaker normalization method is presented and compared to a well known vocal tract length normalization method. With this method, acoustic observations of training and testing speakers are mapped into a normalized acoustic space through speaker-specific transformations with the aim of reducing inter-speaker acoustic variability. For each speaker, an affine transformation is estimated with the goal of reducing the mismatch between the acoustic data of the speaker and a set of target hidden Markov models. This transformation is estimated through constrained maximum likelihood linear regression and then applied to map the acoustic observations of the speaker into the normalized acoustic space. Recognition experiments made use of two corpora, the first one consisting of adults’ speech, the second one consisting of children’s speech. Performing training and recognition with normalized data resulted in a consistent reduction of the word error rate with respect to the baseline systems trained on unnormalized data. In addition, the novel method always performed better than the reference vocal tract length normalization method.

In this paper, we describe Parling, a multimedia system for supporting learning of English as a second language (L2). It is devoted to 8-11 year-old primary school children. The idea behind the system is that famous children's literature offers the right motivating and low-anxiety context where users can improve their vocabulary and learn new language structures. The technological core of the system is a speech recognizer that allows implementing automatic pronunciation assessment. Parling is an adaptive system; it features a set of instructional games that change their vocabulary content dynamically based on the user's learning needs. A preliminary usability test of the prototype system gave positive results.

We report on a study whose objectives were first to investigate the interaction of children with a multimedia system designed for learning a foreign language and featuring automatic speech recognition functionality. Second, we investigated if, using such a system, users meet some particular difficulties in dealing with the microphone or in solving tasks which implies voice interaction. The study was based on the Italian version of "Tell me more. Kids: the city", a multimedia system dedicated to children aged 8 to 10. The results of the study indicate that, when learners were involved in tasks where they could freely decide if and how many times to use the vocal interaction modality, they did not hesitate to use this modality. The study also revealed the importance of the form of feedback chosen to report the system's judgment on the learner's pronunciation to her.

Recognition of children's speech was investigated by considering a phone recognition task. Two baseline systems were trained, one for children and one for adults, by exploiting two Italian speech databases. Under matching conditions, for training and recognition performed with data from the same population group, the phone recognition accuracy was 77.30% and 79.43% for children and adults, respectively. It was found that, for many children, recognition results were as good as for adults. However, a higher variability in phone recognition accuracy across speakers was observed for children than for adults. Vocal tract length normalization, under matched and mismatched training and testing conditions, was also investigated. For both adults and children, a performance improvement, with respect to the baseline systems, was observed.

This work deals with some interesting issues that arose when the ITC-irst broadcast news transcription system was applied to transcribe the audio track of historical documentary films. Due to an evident acoustic and linguistic mismatch between the broadcast news and the new application domain, the initial word error rate was of 46.4%. By exploiting a limited amount of manually annotated training data, adaptation of all components of the transcription system was performed, namely the audio partitioner, the acoustic model, and the language model. This permitted to achieve a word error rate of 30%, which makes automatic transcription of documentary films effective for information retrieval applications.

Reports on experiments of porting the ITC-irst Italian broadcast news recognition system to two spontaneous dialogue domains. The trade-off between performance and the required amount of task specific data was investigated. Porting was experimented by applying supervised adaptation methods to acoustic and language models. By using two hours of manually transcribed speech, word error rates of 26.0% and 28.4% were achieved by the adapted systems. Two reference systems, developed on a larger training corpus, achieved word error rates of 22.6% and 21.2%, respectively.

The paper presents the first achievements in the development of a broadcast news transcription system to be applied for the processing of huge audio archives. In particular, the Italian broadcast news corpus under collection is introduced, and the first implemented baseline system is outlined. The baseline system consists of an audio segmentation module and a speech recognizer featuring a recursive Viterbi beam search, a 64k word lexicon, a tree-based trigram LM representation, and MLLR adaptation. The word error rate of the baseline was 20.9% on planned studio speech and 28.8% on the whole test set.

A challenging scenario is addressed in which a hands-free speech recognizer operates in a noisy office environment with incremental model adaptation functionalities. The use of a single far microphone as well as that of a microphone array input are investigated. In a previous work it was shown that the acoustic mismatch, remaining after the application of microphone array processing, can be further reduced by conditioning hidden Markov models to operating acoustic conditions. Conditioned HMMs are models trained using the filtered version of a clean corpus, which is speech material better representing noisy real environments. Afterwards, conditioned models are used as initial models for unsupervised incremental adaptation. Experimental results of connected digit recognition show that models trained with filtered clean speech allows to obtain better recognition performance than models trained with clean speech. Furthermore, results show a significant performance increase when incremental adaptation is applied, even after recognition of few utterances.

This paper presents some recent improvements in automatic transcription of Italian broadcast news obtained at ITCirst. A first preliminary activity was carried out in order to develop a suitable speech corpus for the Italian language. The resulting corpus, formed by recordings covering 30 hours of radio news, was exploited for developing a baseline system for transcription of broadcast news. The system performs in different stages: acoustic segmentation and classification, speaker clustering, acoustic model adaptation and speech decoding. Major recent advances allowing performance improvement concern with speech segmentation and clustering, acoustic modeling, acoustic model adaptation and the language model.

This paper addresses the problem of hands-free speech recognition in a noisy office environment. An array of six omnidirectional microphones and a corresponding time delay compensation module are used to provide a beamformed signal as input to a HMM-based recognizer. Training of HMMs is performed either using a clean speech database or using a filtered version of the same database. Filtering consists in a convolution with the acoustic impulse response between the speaker and microphone, to reproduce the reverberation effect. Background noise is summed to provide the desired SNR. The paper shows that the new models trained on these data perform better than the baseline ones. Furthermore, the paper investigates on maximum likelihood linear regression (MLLR) adaptation of the new models. It is shown that a further performance improvement is obtained, allowing to reach a 98.7% WRR in a connected digit recognition task, when the talker is at 1.5 m distance from the array.

Feature vector normalization has been successfully used to improve the noise robustness of speech recognizers. Unfortunately, it may cause additional insertion errors in connected digit recognition in clean environments. We propose two methods to reduce the number of insertions. Based on estimated instantaneous signal-to-noise ratio we form a reliability measure for the recognized digits. We discard unreliable digits from the beginning and the end of the recognized digit sequence. Since the proposed reliability hypotheses are independent of the likelihoods produced by an HMM classifier, we are capable of bringing new useful information into the classification process. In addition, we constrain the normalization process on the basis of statistics obtained from the training data. Experimental results show that we are capable of achieving an average 32% string level error rate reduction in simulations of a noisy car environment.

A scenario concerning hands-free connected digit recognition in a noisy office environment is investigated. An array of six omnidirectional microphones and a corresponding time delay compensation module are used to provide a beamformed signal as input to a hidden Markov model (HMM) based recognizer. Two different techniques of phone HMM adaptation have been considered, to reduce the mismatch between training and test conditions. Adaptation material and test material were collected in two different sessions. Results show that a digit accuracy close to 98% can be achieved when the talker is at 1.5 m distance from the array. This result has to be compared with 99.5% accuracy obtained by using a close-talk microphone.

The use of a microphone array for hands-free continuous speech recognition in noisy and reverberant environment is investigated. An array of eight omnidirectional microphones was placed at different angles and distances from the talker. A time delay compensation module was used to provide a beamformed signal as input to a hidden Markov model (HMM) based recognizer. A phone HMM adaptation, based on a small amount of phonetically rich sentences, further improved the recognition rate obtained by applying only beamforming. These results were confirmed both by experiments conducted in a noisy and reverberant environment and by simulations. In the latter case, different conditions were recreated by using the image method to reproduce synthetic versions of the array microphone signals.

In this work we describe the development and evaluation of SpeeData, a prototype for multilingual spoken dataentry. The SpeeData project aims at developing a demonstrator that provides a user-friendly interface for spoken data-entry in two languages: Italian and German. A real world application domain is considered, which is the Land Register of an Italian region in which both languages are officially spoken. Original topics of this paper are the interaction modality for spoken data-entry, the evaluation of a data-entry system, bilingual speech recognition, bilingual speaker adaptation. 1. INTRODUCTION Data-entry can be particularly costly when non electronic information - e.g. contained in documents or pictures - has to be interpreted by a domain expert before being stored into the computer - e.g. medical reporting, diagnostics, cataloging, etc. SpeeData [1] aims at exploring how to gain efficiency in this task by employing state-ofthe -art ASR technology. The ideal scenario would b...

In this work hands-free continuous speech recognition based on microphone arrays is investigated. A set of experiments was carried out using arrays having different numbers of omnidirectional microphones as well as different configurations. Both real and simulated array signals, generated by means of the image method, were used. An enhanced input to a recognizer based on Hidden Markov Models was obtained by a time delay compensation module providing a beamformed signal. HMM adaptation was used to realign the recognizer acoustic modeling to the given acoustic condition.

This paper introduces bootstrap error estimation for automatic tuning of parameters in combined networks, applied as front-end preprocessors for a speech recognition system based on hidden Markov models. The method is evaluated on a large-vocabulary (10 000 words) continuous speech recognition task. Bootstrap estimates of minimum mean squared error allow selection of speaker normalization models improving recognition performance. The procedure allows a flexible strategy for dealing with inter-speaker variability without requiring an additional validation set. Recognition results are compared for linear, generalized radial basis functions and multi-layer perceptron network architectures.

In this paper we present a m ultilingual application for speech t echnology.T h e SpeeData project aims at building a d emonstrator that provides a user-friendly interface for spoken data-entry in two languages: Italian and German.The a p plication domain is the land register of an Italian region in which b o t h languages are ocially spoken.The considered data-entry task is particularly challenging as it considers many dierent t ypes of data -e.g.long t exts, numbers, proper names, tables, etc.-and a v ariety of of pronunciations, since dialects are present a n d users will not always speak in their native language.

This paper describes advances in the use of HMM based technology for speaker independent continuous speech recognition, in noisy environment, under hands free interaction mode. For this purpose an array of four omnidirectional microphones is employed as the acquisition system. The processing of phase information in the cross-power spectrum provides the capability both of locating the talker position and of reconstructing an enhanced speech spectrum. Two enhancement techniques are described, that provide recognition improvement in the case of clean input speech as well as under different adverse conditions. The results refer to the use of a new multichannel corpus, collected in a real environment by a microphone array as well as a close-talk microphone.

The objective of this paper is to describe the activity that is being carried out at IRST laboratories for the development of an HMM-based speaker independent continuous speech recognition system for the Italian language. The recognition system is trained and tested using the acoustic-phonetic continuous speech portion of the APASCI corpus. Acoustic modeling is based on the use of Continuous Density HMMs with gaussian mixture observation densities. As a baseline, a set of 38 Context Independent Units was evaluated using different numbers of mixture components. Then, two other classes of Context Dependent Unit sets were considered, that provide different performance and system complexity. Performance, expressed in terms of Phone loop recognition accuracy and Word loop recognition accuracy, shows an improvement using both of these classes of unit sets, with respect to the baseline. I. INTRODUCTION A baseline of a speaker independent continuous speech recognition system for the Italian ...

Mismatch in training and testing conditions reduces considerably the performance of a speaker-independent HMM-based continuous speech recognizer. Compensation of this mismatch can avoid the complex and time-consuming retraining of the recognizer. This paper describes an acquisition system based on a four omnidirectional microphone array that was employed to reproduce a "beamformed" version of the original acoustic messages acquired in a noisy and reverberant environment, with a talker-microphone distance of one meter. In this preliminary activity, some simple noise compensation techniques (i.e. a Mean Spectrum based Enhancement and a Cepstrum Mean Subtraction) were incorporated in this preprocessing stage to obtain an enhanced version of the given utterance. Feeding a clean-condition trained continuous speech recognizer with enhanced signals led to a significant improvement of performance, if compared to the use of unprocessed single-microphone signals as input. I. INTRODUCTION Perfo...

An architecture for speech recognition is proposed, based on four stages: (1) recognition of the most likely phone sequence using centisecond Hidden Markov Models (HMMs); (2) phone-based lexical and syntactical forward decoding; (3) A phone-based backward pass, producing a Word Hypothesis Structure (WHS); (4) accurate rescoring of the search sub-space represented by the WHS using centisecond HMMs. Experiments carried out on two different tasks show that a recognizer based on the proposed four-stage architecture is able to achieve comparable performance respect to a classic one-stage recognizer. Experimental results show also that the same recognition performance can be obtained with WHSs built with this approach and WHSs built using centisecond HMMs with a potential speed-up, in WHS generation, proportional to the average phoneme duration in centiseconds.

The paper presents a rapid speaker-normalization technique based on neural network spectral mapping. The neural network is used as a front-end of a continuous speech recognition system (speaker(cid:173) dependent, HMM-based) to normalize the input acoustic data from a new speaker. The spectral difference between speakers can be reduced using a limited amount of new acoustic data (40 phonet(cid:173) ically rich sentences). Recognition error of phone units from the acoustic-phonetic continuous speech corpus APASCI is decreased with an adaptability ratio of 25%. We used local basis networks of elliptical Gaussian kernels, with recursive allocation of units and on-line optimization of parameters (GRAN model). For this ap(cid:173) plication, the model included a linear term. The results compare favorably with multivariate linear mapping based on constrained orthonormal transformations.

Stochastic signal models represent a powerful tool for automatic speech recognition. A particular type of stochastic modeling based on first-order hidden Markov models (HMMs), has been increasingly popular, because it has a solid theoretical basis and offers practical advantages. The authors extend the standard HMM theory to parallel hidden Markov models (PHMMs). The parallel model consists of two statistically related HMMs. This configuration has mixture densities of HMM observations whose weights can be made variable depending on the probability of other HMMs being in certain states. This allows one to dynamically adapt observation statistics to acoustic contexts. Some preliminary experiments have been carried out in order to compare the PHMMs with standard HMMs and the results are presented.>

In a large vocabulary isolated word automatic speech recognition system, using Hidden Markov Models (HMM), it is time consuming to perform a detailed likelihood computation of an input utterance for all the words of the dictionary. A tree based A search algorithm is presented for finding the N best word hypotheses, using a new likelihood estimator to drive time asyncronously the search. The proposed likelihood estimator has been tested with two different tasks: a large vocabulary (5000 words) multi-speaker radiological reports recognizer, and a 94 word speaker-independent command language recognizer. In both cases, the speed performance was improved by a factor of about 6, without losing admissibility. Using discrete models, the mean word hypothesis evaluation time is 8 ms on Sun4/330 workstation, the recognition rates vary from 94% to 98% depending on the speaker in the 5000 word task with a bigram language model, and 91% in the speaker-independent task. Introduction A system base...