Rhythm patterns play an important role in the perception of second-language (L2) speech. This paper presents a novel approach to evaluating L2 speech rhythm using low-frequency spectral features inspired by the rhythmogram auditory model. In this paper we investigate several new feature sets for use in training rhythm-centric acoustic models. By capturing information over suprasegmental linguistic units appropriate for rhythmic analysis (including syllables and prosodic feet), these novel features can outperform traditional features in detecting rhythm errors on the ISLE corpus of learner English by 5-15% absolute.

The perception of rhythmic differences among languages relies on varieties in periodicity within prominence groups. But the consensus in phonetic research on rhythm is that existing measures don’t capture true rhythm by that definition instead, they merely measure short-term timing. This work proposes a new rhythm measure, the Generalized Variability Index (GVI), that examines durational contexts over arbitrarily long linguistic distances. To evaluate this new measure, we conducted a set of experiments in automatic language identification using large amounts of data from 11 languages in the Globalphone and TIMIT corpora. When added to baseline rhythm measures, these new GVI features offer absolute improvement in 11-way language classification accuracy by as much as 12%. Moreover, the addition of wider and wider durational context in the GVI continues to contribute information useful for automatic language ID, abating in usefulness only at a distance of about 10 syllables.

Motivated by a desire to assess the prosody of foreign language learners, this study demonstrates the benefit of highlevel syntactic information in automatically deciding where phrase breaks and pitch accents should go in text. The connection between syntax and prosody is well-established, and naturally lends itself to tree-based probabilistic models. With automatically-derived parse trees paired to tree transducer models, we found that categorical prosody tags for unseen text can be determined with significantly higher accuracy than they can with a baseline method that uses n-gram models of part-ofspeech tags. On the Boston University Radio News Corpus, the tree transducer outperformed the baseline by 14% overall for accents, and by 3% overall for breaks. These automatic results fell within this corpus’s range of inter-speaker agreement in assigning accents and breaks to text.

Automatic literacy assessment technology can help children acquire reading skills by providing teachers valuable feedback in a repeatable, consistent manner. Recent research efforts have concentrated on detecting mispronunciations during word-reading and sentence-reading tasks. These token-level assessments are important since they highlight specific errors made by the child. However, there is also a need for more high-level automatic assessments that capture the overall performance of the children. These high-level assessments can be viewed as an interpretive extension to token-level assessments, and may be more perceptually relevant to teachers and helpful in tracking performance over time. In this paper, we model and predict the overall reading ability of young children reading a list of English words aloud. The data consist of audio recordings, collected in real kindergarten to second grade classrooms from children from native English- and Spanish-speaking households. This research is broken into two main parts. The first part is a user study, in which 11 human evaluators rated the children on their overall reading ability based on the audio recordings. The evaluators were volunteers from a diverse background, seven of whom were native speakers of American English and four that were fluent speakers of English as a secondary language. While none of the evaluators were trained reading experts or licensed teachers, a subset of them were linguists and researchers with experience in automatic literacy assessment. As part of this work, we analyzed the effect of the evaluator's background on inter-evaluator agreement. In the second part, we ran machine learning experiments to predict evaluators' scores using features automatically extracted from the audio. The features were human-inspired and correlated with cues human evaluators stated they used: pronunciation correctness, speaking rate, and fluency. We investigated various automated methods to verify the correctness of the word pronunciations and to detect disfluencies in the children's speech using held-out annotated data. Using linear regression techniques, we automatically predicted individual evaluators' high-level scores with a mean Pearson correlation coefficient of 0.828, and we predicted average evaluator's scores with correlation 0.946. Both these human-machine agreement statistics exceeded the mean inter-evaluator agreement statistics.

This paper presents a novel student model intended to automate word-list-based reading assessments in a classroom setting, specifically for a student population that includes both native and nonnative speakers of English. As a Bayesian Network, the model is meant to conceive of student reading skills as a conscientious teacher would, incorporating cues based on expert knowledge of pronunciation variants and their cognitive or phonological sources, as well as prior knowledge of the student and the test itself. Alongside a hypothesized structure of conditional dependencies, we also propose an automatic method for refining the Bayes Net to eliminate unnecessary arcs. Reading assessment baselines that use strict pronunciation scoring alone (without other prior knowledge) achieve 0.7 correlation of their automatic scores with human assessments on the TBALL dataset. Our proposed structure significantly outperforms this baseline, and a simpler data-driven structure achieves 0.87 correlation through the use of novel features, surpassing the lower range of inter-annotator agreement. Scores estimated by this new model are also shown to exhibit the same biases along demographic lines as human listeners. Though used here for reading assessment, this model paradigm could be used in other pedagogical applications like foreign language instruction, or for inferring abstract cognitive states like categorical emotions.

Children need to master reading letter-names and letter-sounds before reading phrases and sentences. Pronunciation assessment of letter-names and letter-sounds read aloud is an important component of preliterate children's education, and automating this process can have several advantages. The goal of this work was to automatically verify letter-names spoken by kindergarteners and first graders in realistic classroom noise conditions. We applied the same techniques developed in our previous work on automatic letter-sound verification by comparing and optimizing different acoustic models, dictionaries, and decoding grammars. Our final system was unbiased with respect to the child's grade, age, and native language and achieved 93.1% agreement (0.813 kappa agreement) with human evaluators, who agreed among themselves 95.4% of the time (0.891 kappa).

Past studies have shown that a native Spanish speaker’s use of phrasal prominence is a good indicator of her level of English prosody acquisition. Because of the cross-linguistic differences in the organization of phrasal prominence and durational contrasts, we hypothesize that those speakers with English-like prominence in their L2 speech are also expected to have acquired English-like rhythm. Statistics from a corpus of native and nonnative English confirm that speakers with an Englishlike phrasal prominence are also the ones who use English-like rhythm. Additionally, two methods of automatic score generation based on vowel duration times demonstrate a correlation of at least 0.6 between these automatic scores and subjective scores for phrasal prominence. These findings suggest that simple vowel duration measures obtained from standard automatic speech recognition methods can be salient cues for estimating subjective scores of prosodic acquisition, and of pronunciation in general.

Phonological transfer is the influence of a first language on phonological variations made when speaking a second language. With automatic pronunciation assessment applications in mind, this study intends to uncover evidence of phonological transfer in terms of articulation. Real-time MRI videos from three German speakers of English and three native English speakers are compared to uncover the influence of German consonants on close English consonants not found in German. Results show that nonnative speakers demonstrate the effects of L1 transfer through the absence of articulatory contrasts seen in native speakers, while still maintaining minimal articulatory contrasts that are necessary for automatic detection of pronunciation errors, encouraging the further use of articulatory models for speech error characterization and detection. Index Terms: real-time MRI, nonnative speech, articulation, phonological transfer

Articulatory Phonology’s link between cognitive speech planning and the physical realizations of vocal tract constrictions has implications for speech acoustic and duration modeling that should be useful in assigning subjective ratings of pronunciation quality to nonnative speech. In this work, we compare traditional phoneme models used in automatic speech recognition to similar models for articulatory gestural pattern vectors, each with associated duration models. What we find is that, on the CDT corpus, gestural models outperform the phonemelevel baseline in terms of correlation with listener ratings, and in combination phoneme and gestural models outperform either one alone. This also validates previous findings with a similar (but not gesture-based) pseudo-articulatory representation. Index Terms: pronunciation modeling, nonnative speech, articulatory phonology

Motivated by potential applications in second-language pedagogy, we present a novel approach to using articulatory information to improve automatic detection of typical phone-level errors made by nonnative speakers of English-a difficult task that involves discrimination between close pronunciations. We describe a reformulation of the hidden-articulator Markov model (HAMM) framework that is appropriate for the pronunciation evaluation domain. Model training requires no direct articulatory measurement, but rather involves a constrained and interpolated mapping from phone-level transcriptions to a set of physically and numerically meaningful articulatory representations. Here, we define two new methods of deriving articulatory-based features for classification: one, by concatenating articulatory recognition results over eight streams representative of the vocal tract's constituents; the other, by calculating multidimensional articulatory confidence scores within these representations based on general linguistic knowledge of articulatory variants. After adding these articulatory features to traditional phone-level confidence scores, our results demonstrate absolute reductions in combined error rates for verification of segment-level pronunciations produced by nonnative speakers in the ISLE corpus by as much as 16%-17% for some target segments, and a 3%-4% absolute improvement overall.

To automatically assess young children’s reading skills as demonstrated by isolated words read aloud, we propose a novel structure for a Bayesian Network classifier. Our network models the generative story among speech recognition-based features, treating pronunciation variants and reading mistakes as distinct but not independent cues to a qualitative perception of reading ability. This Bayesian approach allows us to estimate the probabilistic dependencies among many highly-correlated features, and to calculate soft decision scores based on the posterior probabilities for each class. With all proposed features, the best version of our network outperforms the C4.5 decision tree classifier by 17% and a Naive Bayes classifier by 8%, in terms of correlation with speaker-level reading scores on the Tball data set. This best correlation of 0.92 approaches the expert inter-evaluator correlation, 0.95.

This paper describes the design and realization of an automatic system for assessing and evaluating the language and literacy skills of young children. This system was developed in the context of the TBALL (technology based assessment of language and literacy) project and aims at automatically assessing the English literacy skills of both native talkers of American English and Mexican-American children in grades K-2. The automatic assessments were carried out employing appropriate speech recognition and understanding techniques. In this paper, we describe the system focusing on the role of the multiple sources of information at our disposal. We present the content of the assessment system, discuss some issues in creating a child-friendly interface, and how to provide a suitable feedback to the teachers. In addition, we will discuss the different assessment modules and the different algorithms used for speech analysis.

This study examines methods for recognizing different classes of phones from accented speech based on voice on-set time (VOT). These methods are tested on data from the Tball corpus of Los Angeles-area elementary school chil-dren [1]. The methods proposed and tested are: 1) to train models based on standard English VOT contrasts and then extract the VOT characteristics of the phones by measuring the duration of phone-level and sub-phone-level alignments, 2) to train phone models with explicit aspiration, and 3) to train different models for different phoneme classes of VOT times. Error rates of 23-53 % for different phone classes are reported for the rst method, 5-57 % for the second method, and 0-36 % for the third. The results show that different methods work better on different phone classes. We inter-pret these results in relation to past research on VOT, explain possible uses for these ndings, and propose directions for future research. Index Terms: recognition of speech variation, pronuncia-tion variation, voice onset time.

Arguably the most important part of automatically assessing a new reader's literacy is in verifying his pronunciation of read-aloud target words.But the pronunciation evaluation task is especially difficult in children, non-native speakers, and pre-literates.Traditional likelihood ratio thresholding methods do not generalize easily, and even expert human evaluators do not always agree on what constitutes an acceptable pronunciation.We propose new recognition-and alignment-based features in a decision tree classification framework, along with the use of prior linguistic information and human perceptual evaluations.Our classification methods demonstrate a 91% agreement with the voted results of 20 human evaluators who agree among themselves 85% of the time.

The robust understanding of sarcasm in a spoken dialogue system requires a reformulation of the dialogue manager’s basic assumptions behind, for example, user behavior and grounding strategies. But automatically detecting a sarcastic tone of voice is not a simple matter. This paper presents some experiments toward sarcasm recognition using prosodic, spectral, and contextual cues. Our results demonstrate that spectral and contextual features can be used to detect sarcasm as well as a human annotator would, and confirm a long-held claim in the field of psychology – that prosody alone is not sufficient to discern whether a speaker is being sarcastic. Index Terms: dialogue, user modeling, sarcasm, speech acts

A robust language learning system, designed to help students practice a foreign language along with a machine tutor, must provide meaningful feedback to users by isolating and localizing their pronunciation errors. This paper presents a new technique for automatic syllable stress detection that is tailored for language-learning purposes. Our method, which uses basic prosodic features and others related to the fundamental frequency slope and RMS energy range, is at least as accurate as an expert human listener, but requires no human supervision other than a pre-defined dictionary of expected lexical stress patterns for all words in the system's vocabulary. Optimal feature choices exhibited an 87-89% accuracy compared with human-tagged stress labels, exceeding the inter-human agreement commonly held to be about 80%.