A number of recent studies have focused on the conceptualized expression of emotions as a three-dimensional space. This paper proposes a new approach to emotion recognition, making use of two of the emotional dimensions and their relationship with different kinds of features. The main idea consists in associating prosodic features, derived from pitch, loudness, and duration, with the activation or arousal dimension, and quality features, i.e. phonation type, articulation manner, voice timbre, with the evaluation and pleasure dimension, in a way that different classification methods can be applied for each specific case. Most important results are achieved for the speaker-independent case and three classes, with a recognition rate close to 80%. 1.

In order to detect misrecognitions that may result from a mismatch between training and testing data, we use a confidence measure (CM) that collects a set of features during recognition and from the N-best list that is output by the recognizer. A neural network (NN) then calculates the probability that the utterance was recognized correctly based on these features. Since for misrecognized utterances the resulting phoneme alignments are often erroneous, we introduced some new features that are based on phoneme durations. The durations found by the recognizer are compared to the durations present in the training data base and the results of these comparisons serve as input for the NN. A great advantage of the duration-related features is that they are independent of the recognizer in contrast to e.g. acoustic scorebased features. We also use some score-related features that have proven to be useful in the past. Simultaneously with determining the confidence for a recognition result, we try to detect if in case of a misrecognition the utterance was an out of vocabulary (OOV) utterance. Using the complete set of 46 features we can achieve a correct classification rate of 90%. The word error rate can be reduced by 92% at a false rejection rate of 5.1% on a test task that consists of 35 speakers and includes more than 50% OOV utterances. OOV words were detected correctly in 91% of the cases. The presented CM is also used in a semi-supervised speaker adaptation scheme.

We show how prosody can be used in speech understanding systems. This is demonstrated with the VERBMOBIL speech to-speech translation system which, to our knowledge, is the first complete system which successfully uses prosodic information in the linguistic analysis. Prosody is used by computing probabilities for clause boundaries, accentuation, and different types, of sentence mood for each of the word hypotheses computed by the word recognizer. These probabilities guide the search of the linguistic analysis. Disambiguation is already achieved during the analysis and not by a prosodic verification of different linguistic hypotheses. So far, the most useful prosodic information is provided by clause boundaries. These are detected with a recognition rate of 94%. For the parsing of word hypotheses graphs, the use of clause boundary probabilities yields a speed-up of 92% and a 96% reduction of alternative readings.

This paper presents an approach for fast, unsupervised, online MLLR speaker adaptation using two MAP-like weighting schemes, a static and a dynamic one. While for the standard MLLR approach several sentences are necessary before a reliable estimation of the transformations is possible, the weighted approach shows good results even if adaptation is conducted after only a few short utterances. Experimental results show that using the static approach can improve the word error rate by approx. 27% if adaptation is conducted after every 4 utterances (single words or short phrases). Using the dynamic approach, results can be improved by 28%. The most important advantage of the dynamic weight is that it is rather insensitive with respect to the initial weight whereas for the static approach it is very critical which initial weight to chose. Moreover, useful values for the weights in the static case depend very much on the corpus. If the standard MLLR approach is used, even a drastic increase in sentence error rate can be observed for these small amounts of adaptation data.

Parsing can be improved in automatic speech understanding if prosodic boundary marking is taken into account, because syntactic boundaries are often marked by prosodic means. Because large databases are needed for the training of statistical models for prosodic boundaries, we developed a labeling scheme for syntactic-prosodic boundaries within the German Verbmobil project (automatic speech-to-speech translation). We compare the results of classifiers (multi-layer perceptrons and language models) trained on these syntactic-prosodic boundary labels with classifiers trained on perceptual-prosodic and purely syntactic labels. Recognition rates of up to 96% were achieved. The turns that we need to parse consist of 20 words on the average and frequently contain sequences of partial sentence equivalents due to restarts, ellipsis, etc. For this material, the boundary scores computed by our classifiers can successfully be integrated into the syntactic parsing of word graphs; currently, they improve the parse time by 92% and reduce the number of parse trees by 96%. This is achieved by introducing a special prosodic syntactic clause boundary (PSCB) symbol into our grammar and guiding the search for the best word chain with the prosodic boundary scores.

We present the prosody module of the VERBMOBIL speech-to-speech translation system, the world wide first complete system, which successfully uses prosodic information in linguistic analysis. This is achieved by computing probabilities for clause boundaries, accentuation, and different types of sentence mood for each of the word hypotheses computed by the word recognizer. These probabilities guide the search of the linguistic analysis. Disambiguation is already achieved during the analysis and not by a prosodic verification of different linguistic hypotheses. So far, the most useful prosodic information is provided by clause boundaries. These are detected with a recognition rate of 94%. For the parsing of word hypotheses graphs, the use of clause boundary probabilities yields a speed-up of 92% and a 96% reduction of alternative readings.

In this paper, we give a first account of speech tempo and its change in spontaneous speech in a very large data base (Verbmobil, i.e., human-human appointment dialogs). As features representing speech tempo, we computed mean normalized speech duration (speaking rate) and normalized phone duration in different ways. The importance of these features is evaluated with an automatic classification of boundaries and accents where different sets of prosodic features (including also information about F0, energy, pause, etc.) were used. The best results (83% for accents, 88% for boundaries, two classes each) could be achieved when all features were used. For the 2nd issue change of tempo was labelled manually. We present the characterizing feature values for changes from slow to fast and from fast to slow, as well as the results of an automatic classification of change of tempo (72% for three classes). Finally, we discuss the possible function of change of tempo and its use in automatic speech processing. (orig.)

This paper presents an integrated approach for the segmentation and classification of dialog acts (DA) in the Verbmobil project. In Verbmobil it is often sufficient to recognize the sequence of DAs occurring during a dialog between the two partners. In our previous work we segmented and classified a dialog in two steps: first we calculated hypotheses for the segment boundaries and decided for a boundary if the probabilities exceeded a predefined threshold level. Second we classified the segments into DAs using semantic classification trees or stochastic language models. In our new approach we integrate the segmentation and classification in the A*-algorithm to search for the optimal segmentation and classification of DAs on the basis of word hypotheses graphs (WHGs). The hypotheses for the segment boundaries are calculated with the help of a stochastic language model operating on the word chain and a multi-layer perceptron (MLP) classifying prosodic feature. The DA classification is done using a category based language model for each DA. For our experiments we used data from the Verbobil-corpus. (orig.)

We propose multilevel semantic classification trees to combine different information sources for predicting speech events (e.g. word chains, phrases, etc.). Traditionally in speech recognition systems these information sources (acoustic evidence, language model) are calculated independently and combined via Bayes rule. The proposed approach allows one to combine sources of different types it is no longer necessary for each source to yield a probability. Moreover the tree can look at several information sources simultaneously. The approach is demonstrated for the prediction of prosodically marked phrase boundaries, combining information about the spoken word chain, word category information, prosodic parameters, and the result of a neural network predicting the boundary on the basis of acoustic-prosodic features. The recognition rates of up to 90% for the two class problem boundary vs. no boundary are already comparable to results achieved with the above mentioned Bayes rule approach that combines the acoustic classifier with a 5-gram categorical language model. This is remarkable, since so far only a small set of questions combining information from different sources have been implemented.

We describe a number of experiments that demonstrate the usefulness of prosodic information for a processing module which parses spoken utterances with a feature-based grammar employing empty categories.We show that by requiring certain prosodic properties from those positions in the input, where the presence of an empty category has to be hypothesized, a derivation can be accomplished more eciently.The approach has been implemented in the machine translation project Verbmobil and results in a signicant reduction of the work-load for the parser.

In natural languages, the words within an utterance are often correlated over large distances. Long-spanning contextual effects of this type cannot be efficiently and robustly captured by the traditional N-gram approaches of stochastic language modelling. We present a new kind of stochastic grammar --- the permugram model. A permugram model is obtained by linear interpolation of a large number of conventional bigram, trigram, or polygram models which operate on different permutations of the input word sequence under consideration. This way, stochastic dependences between word pairs or word triples lying adjacent as well as remote in the input text can be captured simultaneously without the requirement of very large N-grams. Using the permugram model, we achieved test set perplexity reductions of 5--10% compared with interpolated N-gram models, depending on the application. 1. INTRODUCTION In natural languages, the words within an utterance are often correlated over large distances; fo...

A large corpus has been created automatically and read by 100 speakers. Phrase boundaries were labeled in the sentences automatically during sentence generation. Perception experiments on a subset of 500 utterances showed a high agreement between the automatically generated boundary markers and the ones perceived by listeners. Gaussian distribution and polynomial classifiers were trained on a set of prosodic features computed from the speech signal using the automatically generated boundary markers. Comparing the classification results with the judgments of the listeners yielded in a recognition rate of 87%. A combination with stochastic language models improved the recognition rate to 90%. We found that the pause and the durational features are most important for the classification, but that the influence of F0 is not neglectable.>

In written language, punctuation is used to separate main and subordinate clause.In spoken language, ambiguities arise due to missing punctuation, but clause boundaries are often marked prosodically and can be used instead.We detect PCBs (Prosodically marked Clause Boundaries) by using prosodic features (duration, intonation, energy, and pause information) with a neural network, achieving a recognition rate of 82%.PCBs are integrated into our grammar using a special syntactic category 'break' that can be used in the phrase-structure rules of the grammar in a similar way as punctuation is used in grammars for written language.Whereas punctuation in most cases is obligatory, PCBs are sometimes optional.Moreover, they can in principle occur everywhere in the sentence due e.g. to hesitations or misrecognition.To cope with these problems we tested two different approaches: A slightly modified parser for word chains containing PCBs and a word graph parser that takes the probabilities of PCBs into account.Tests were conducted on a subset of infinitive subordinate clauses from a large speech database containing sentences from the domain of train table inquiries.The average number of syntactic derivations could be reduced by about 70 % even when working on recognized word graphs.

In this paper a method for the automatic labeling of phrase accents is described, based on a large text corpus that has been generated automatically and read by 100 speakers.Perception experiments on a subset of 500 utterances show a high agreement between the automatically generated accent labels and the judgment scores obtained.We computed different prosodic feature vectors from the speech signal for each syllable and trained different Gaussian distribution classifiers and artificial neural networks using the automatically generated accent labels.Recognition rates of up to 83% could be achieved for the distinction of accentuated vs. unaccentuated syllables.Similar results could be obtained for the comparison of the listeners judgments with the automatic classification.

In this paper we present a new method transforming speech signals to voice source signals (VSS) using articial neural networks (ANN).We will point out that the ANN mapping of speech signals into source signals is quite accurate, and most of the irregularities in the speech signal will lead to an irregularity in the source signal, produced by the ANN (ANN-VSS).We will show that the mapping of the ANN is robust with respect to untrained speakers, di erent recording conditions and facilities, and di erent v ocabularies.We will also present preliminary results which show that from the ANN source signal pitch periods can be determined accurately.

In this paper rst experiments with naive persons using the speech understanding and dialog system EVAR are discussed.The domain of EVAR is train table inquiry.We observed that in real human-human dialogs when the o cer transmits the information the customer very often interrupts.Many of these interruptions are just repetitions of the time of day given by the o cer.The functional role of these interruptions is determined b y prosodic cues only.An important result of the experiments with EVAR is that it is hard to follow the system giving the train connection via speech synthesis.In this case it is even more important than in human-human dialogs that the user has the opportunity to interact during the answer phase.Therefore we extended the dialog m o dule to allow the user to repeat the time of day and we added a p r osody module guiding the continuation of the dialog.

A new algorithm for the determination of fundamental frequency (F/sub 0/) contours is presented. For each voiced frame appropriate divisors of the frequency with the maximum energy in the spectrum are taken as F/sub 0/ candidates. An F/sub 0/ contour is computed using a dynamic programming (DP) method by minimizing a weighted sum of the difference between consecutive candidates and the distance of the candidates to a predetermined local target value. With this algorithm a coarse error rate of 0.6% on the frame level and of 6.4% on the sentence level is achieved on a German speech database. On the average the difference to the reference is 1.9 Hz. The algorithm outperforms two conventional algorithms tested on the same data.>

This article describes the dialog module of EVAR. The dialog is seen as a sequence of dialog acts uttered by the system and the user. From a corpus of real humanhuman dialogs a model was extracted. The model covers all sequences of dialog acts observed in the corpus. Each dialog act is modelled as a set of pragmatic, semantic and syntactic concepts. The properties of the concepts and the current dialog state are used to identify the actual dialog act. Each user utterance is interpreted by the system and the system reacts with the appropriate dialog act. After the user has defined his request completly, the system starts a database query and finally generates a synthesized natural language answer. The answer generation is realised with sentence masks, where the information from the database enquiry is filled in at the appropriate slots. A dialog memory which allows the interpretation of elliptical sentences and proforms is updated after each utterance.

The integration of multilayered and recurrent artificial neural networks (ANNs) with hidden Markov models (HMMs) is addressed. ANNs are suitable for approximating functions that compute new acoustic parameters, whereas HMMs have been proven successful at modeling the temporal structure of the speech signal. In the approach described, the ANN outputs constitute the sequence of observation vectors for the HMM. An algorithm is proposed for global optimization of all the parameters. Results on speaker-independent recognition experiments using this integrated ANN-HMM system on the TIMIT continuous speech database are reported.

Dans le cadre d'un decodeur acoustique-phonetique hybride (ANN/HMM), trois reseaux de neurones (ANNs) specialises ont ete developpes et evalues. Un de ces ANNs detecte le mode d'articulation. Les deux autres ANNs decrivent le signal en termes du lieu d'articulation. Un reseau est utilise pour classifier les consonnes nasales et plosives. Un autre est utilise pour la classification des fricatives Le design de ces reseaux est inspire par des connaissances acoustiques et phonetiques. Les entrees, la topologie et le codage des sorties ont ete optimises pour chacun des reseaux

In this paper we compare two hybrid acoustic-phonetic decoders based on Artificial Neural Networks (ANN). We evaluate them on the task of recognizing stop phones in continuous speech independently from the speaker. ANNs are well suited to perform detailed phonetic distinctions. In general, techniques based on Dynamic Programming (DP), in particular Hidden Markov Models (HMMs), have proven to be successful at modeling the temporal structure of the speech signal. In the approach described here, the ANN outputs constitute the sequence of observation vectors for the HMM. An algorithm is proposed for global optimization of all the parameters of the ANN/HMM decoder. Comparative experiments using this ANN/HMM hybrid decoder and another ANN-DP hybrid are reported for the TIMIT database. 1 Introduction Artificial Neural Networks (ANNs) effectively perform phonetic classification, but have not proven yet to model the temporal structure of the speech signal reasonably well [Lip89, Rob90, Ben90b...