The presence of disfluencies in spontaneous speech, while poses a challenge for robust automatic recognition, also offers means for gaining additional insights into understanding a speaker's communicative and cognitive state. This paper analyzes disfluencies in children's spontaneous speech, in the context of spoken dialog based computer game play, and addresses the automatic detection of disfluency boundaries. Although several approaches have been proposed to detect disfluencies in speech, relatively little work has been done to utilize visual information to improve the performance and robustness of the disfluency detection system. This paper describes the use of visual information along with prosodic and language information to detect the presence of disfluencies in a child's computer-directed speech and shows how these information sources can be integrated to increase the overall information available for disfluency detection. The experimental results on our children's multimodal dialog corpus indicate that disfluency detection accuracy of over 80% can be obtained by utilizing audio-visual information. Specifically, results showed that the addition of visual information to prosody and language features yield relative improvements in disfluency detection error rates of 3.6% and 6.3%, respectively, for information fusion at the feature level and decision level.

Recent studies in our lab show that emotions in speech are manifested as, besides supra-segmental trends, distinct variations in phoneme-level prosodic and spectral parameters. In this paper, we further investigate the significance of this finding in the context of emotional speech synthesis. Specifically, we study phoneme-level signal property manipulation in transforming the emotional information conveyed in a speech utterance. We analyze the effect of individual and combined modifications of F0, duration, energy and spectrum using data recorded by a professional actress with happy, angry, sad and neutral expressiveness. We use content matched source-target pairs and apply TDPSOLA for prosody and LPC for spectrum modifications by directly extracting the required parameters from the target speech. Listening tests conducted with 10 naive raters show that modification of prosody and spectral envelope parameters by themselves is not sufficient. However, when applied together, modifying spectrum and prosody at the phone level gives successful results for most emotion pairs, except conversion to happy targets. We also observe that at the phoneme level, spectral envelope modifications are more effective than local prosodic modifications; and that, duration modifications are more effective than pitch modifications. The results confirm our hypothesis that phoneme level modifications can be used to fine tune the ensuing suprasegmental-parameter-based modifications to improve the overall quality of synthesized emotions.

Few studies exist on the topic of emotion encoding in speech in the articulatory domain. In this report, we analyze articulatory data collected during simulated emotional speech production and investigate differences in speech articulation among four emotion types; neutral, anger, sadness and happiness. The movement data of the tongue tip, the jaw and the lower lip, along with speech, were obtained from a subject using an Electromagnetic articulography (EMA) system. The effectiveness of the articulatory parameters in emotion classification was also investigated. A general articulatory behavior observed was that emotionally elaborated speech production exhibits more peripheral articulations when compared to neutral speech. The tongue tip, jaw and lip positioning become more advanced when emotionally charged. This tendency was especially prominent for the tongue tip and jaw movements associated with sad speech. Angry speech was characterized by greater ranges of displacement and velocity, while it was opposite for sad speech. Happy speech was comparable in articulation to the neutral speech, but showed the widest range of pitch variation. It, however, remains to be seen if there is a trade-off between articulatory activity and voicing activity in emotional speech production. Multiple discriminant analysis showed that emotion is better classified in the articulatory domain. One probable reason is that the independency in the manipulation of each articulator may provide more degrees of freedom and less overlap in the articulatory parameter space. Analysis also showed distinct emotion effects for different phonemes: the high front vowel /IY/ was found to be less discriminated in both articulatory and acoustic domains than other peripheral vowels such as /AA/ and /UW/. It is likely that the physical boundary effect in the /IY/ articulation may leave less room to vary the tongue positioning and/or the lip configuration when compared to other vowels, resulting in less acoustic contrast among emotion types.

In this study, we investigate politeness and frustration behavior of children during their spoken interaction with computer characters in a game. We focus on automatically detecting frustrated, polite and neutral attitudes from the child’s speech (acoustic and language) communication cues and study their differences as a function of age and gender. The study is based on a Wizard-of-Oz dialog corpus of 103 children playing a voice activated computer game. Statistical analysis revealed that there was a significant gender effect on politeness with girls in this data exhibiting more explicit politeness markers. The analysis also showed that there is a positive correlation between frustration and the number of dialog turns reflecting the fact that longer time spent solving the puzzle of the game led to a more frustrated child. By combining acoustic and language cues for the task of automatic detection of politeness and frustration, we obtain average accuracy of 84.7 % and 71.3%, respectively, by using age dependent models and 85 % and 72%, respectively, for gender dependent models. 1.

The interaction between human beings and computers will be more natural if computers are able to perceive and respond to human non-verbal communication such as emotions. Although several approaches have been proposed to recognize human emotions based on facial expressions or speech, relatively limited work has been done to fuse these two, and other, modalities to improve the accuracy and robustness of the emotion recognition system. This paper analyzes the strengths and the limitations of systems based only on facial expressions or acoustic information. It also discusses two approaches used to fuse these two modalities: decision level and feature level integration. Using a database recorded from an actress, four emotions were classified: sadness, anger, happiness, and neutral state. By the use of markers on her face, detailed facial motions were captured with motion capture, in conjunction with simultaneous speech recordings. The results reveal that the system based on facial expression gave better performance than the system based on just acoustic information for the emotions considered. Results also show the complementarily of the two modalities and that when these two modalities are fused, the performance and the robustness of the emotion recognition system improve measurably.

Recognizing human emotions/attitudes from speech cues has gained increased attention recently. Most previous work has focused primarily on suprasegmental prosodic features calcu-lated at the utterance level for modeling against details at the segmental phoneme level. Based on the hypothesis that dif-ferent emotions have varying effects on the properties of the different speech sounds, this paper investigates the usefulness of phoneme-level modeling for the classification of emotional states from speech. Hidden Markov models (HMM) based on short-term spectral features are used for this purpose using data obtained from a recording of an actress ’ expressing 4 different emotional states- anger, happiness, neutral, and sadness. We designed and compared two sets of HMM classifiers: a generic set of “emotional speech ” HMMs (one for each emotion) and a set of broad phonetic-class based HMMs for each emotion type considered. Five broad phonetic classes were used to explore the effect of emotional coloring on different phoneme classes, and it was found that spectral properties of vowel sounds were the best indicator of emotions in terms of the classification per-formance. The experiments also showed that the better per-formance can be obtained by using phoneme-class classifiers than generic “emotional ” HMM classifier and classifiers based on global prosodic features. To see the complementary effect of the prosodic and spectral features, the two classifiers were combined at the decision level. The improvement was 0.55% in absolute (0.7 % relatively) compared with the result from phoneme-class based HMM classifier. 1.

Understanding the fine details of children’s speech and their gestural characteristics helps, among other things, in creating natural computer interfaces. We analyze reference marking in young children’s computer-directed speech using audio-video data from 3to 6-year-old children engaged in a series of age-appropriate computer tasks, using aWizard of Oz technique. Along with speech transcriptions and acoustic information, discourse (referential devices, conversational repairs) and gestural characteristics (hand/head movement type) were annotated in a synchronized multi-layer system. The results point to the developmental variability and the multimodal nature of the speech young children produce while interacting with the computer agent, suggesting that interfaces addressed to this age group should be specifically designed to integrate multisensory information as well as to adjust to the child’s specific needs and interactional style.

In this study, we investigate acoustic properties of speech associ-ated with four different emotions (sadness, anger, happiness, and neutral) intentionally expressed in speech by an actress. The aim is to obtain detailed acoustic knowledge on how speech is modulated when speaker’s emotion changes from neutral to a certain emotional state. It is based on measurements of acoustic parameters related to speech prosody, vowel articulation and spectral energy distribution. Acoustic similarities and differences among the emotions are then explored with mutual information computation, multidimensional scaling, and comparison of acoustic likelihoods relative to the neu-tral emotion. In addition, acoustic separability of the emotions is tested using the discriminant analysis at the utterance level and the result is compared with human evaluation. Results show that hap-piness/anger and neutral/sadness share similar acoustic properties in this speaker. Speech associated with anger and happiness are characterized by longer utterance duration, shorter inter-word si-lence, higher pitch and energy values with wider ranges, showing the characteristics of exaggerated or hyperarticulated speech. The discriminant analysis indicates that within-group acoustic separa-bility is relatively poor, suggesting that conventional acoustic pa-rameters examined in this study are not effective in describing the emotions along the valence (or pleasure) dimension. It is noted that RMS energy, inter-word silence and speaking rate are useful in dis-tinguishing sadness from others. Interestingly, the between-group difference in formant patterns seems better reflected in back vowels such as /a / (/father/) than in the front vowels. Larger lip opening and/or more tongue constriction at the mid or rear part of the vocal tract could be underlying reasons. 1.

Developmental changes in the human speech production system signal age-dependent variability in the speech signal properties. In this paper, an information-theoretic analysis of developmental changes in the speech signal is presented. The effects of age and signal bandwidth on speech signal features are analyzed especially motivated by implications to automatic recognition of children's speech. Mutual information is calculated between cepstral features and the vowel phonetic class for different age groups and signal bandwidths. The results show that information contained in cepstral features about phonetic classes increases as bandwidth increases for all ages. Cepstral features of adult speech convey more information compared to that of children's speech for both genders. Information increases rapidly between bandwidths 500 Hz and 4500 Hz. These findings based on mutual information correspond well with vowel recognition (classification) experiments. The vowel recognition experiment shows that as bandwidth increases recognition accuracy increases as well.