Pronunciation error patterns (EPs) are patterns of mispronunciation frequently produced by language learners, and are usually different for different pairs of target and native languages. Accurate information of EPs can offer helpful feedbacks to the learners to improve their language skills. However, the major difficulty of EP detection comes from the fact that EPs are intrinsically similar to their corresponding canonical pronunciation, and different EPs corresponding to same canonical pronunciation are also intrinsically similar to each other. As a result, distinguishing EPs from their corresponding canonical pronunciation and between different EPs of the same phoneme is a difficult task-perhaps even more difficult than distinguishing between different phonemes in one language. On the other hand, the cost of deriving all EPs for each pair of target and native languages is high, usually requiring extensive expert knowledge or high-quality annotated data. Unsupervised EP discovery from a corpus of learner recordings would thus be an attractive addition to the field. In this paper, we propose new frameworks for both supervised EP detection and unsupervised EP discovery. For supervised EP detection, we use hierarchical multi-layer perceptrons (MLPs) as the EP classifiers to be integrated with the baseline using HMM/GMM in a two-pass Viterbi decoding architecture. Experimental results show that the new framework enhances the power of EP diagnosis. For unsupervised EP discovery we propose the first known framework, using the hierarchical agglomerative clustering (HAC) algorithm to explore sub-segmental variation within phoneme segments and produce fixed-length segment-level feature vectors in order to distinguish different EPs. We tested K-means (assuming a known number of EPs) and the Gaussian mixture model with the minimum description length principle (estimating an unknown number of EPs) for EP discovery. Preliminary experiments offered very encouraging results, although there is still a long way to go to approach the performance of human experts. We also propose to use the universal phoneme posteriorgram (UPP), derived from an MLP trained on corpora of mixed languages, as frame-level features in both supervised detection and unsupervised discovery of EPs. Experimental results show that using UPP not only achieves the best performance, but also is useful in analyzing the mispronunciation produced by language learners.

Supervised approaches can learn a spoken document summarizer generating high-quality summaries using a set of training examples matched to the domain of target documents. However, preparing a sufficient number of in-domain training examples is expensive. In this paper we propose an approach for unsupervised domain adaptation for spoken document summarization, so no in-domain training examples are needed. A summarizer is first learned from a set of out-of-domain training examples by a supervised summarization approach based on structured support vector machine, and this summarizer is used to generate a set of initial summaries for the target spoken documents. The target documents and their initial machine-generated summaries then serve as extra training examples for learning a new summarizer, which further updates the summaries of the target spoken documents. This process is continued iteratively to incrementally improve the summarizer for the target spoken documents. Moreover, extra approaches transforming the feature representations based on the data distribution in the target domain and augmenting the representations with an extra set of domain-specific features are also proposed. Encouraging results were obtained in summarizing Mandarin-English code-switching course lectures using training examples from Mandarin broadcast news.

We propose a framework for computer-assisted language learning as a pedagogical dialogue game. The goal is to offer personalized learning sentences on-line for each individual learner considering the learner's learning status, in order to strike a balance between more practice on poorly-pronounced units and complete practice on the whole set of pronunciation units. This objective is achieved using a Markov decision process (MDP) trained with reinforcement learning using simulated learners generated from real learner data. Preliminary experimental results on a subset of the example dialogue script show the effectiveness of the framework.

In Computer-Aided Pronunciation Training, we hope to specify the type of mispronunciation, or Error Pattern (EP), the language learner has made as a more effective feedback. But derivation of EPs usually requires expert knowledge and pedagogical experiences, which is not easy to obtain for each pair of target and native languages. In this paper we propose a preliminary framework toward unsupervised discovery of EPs from a corpus of learners' recordings. We use Universal Phoneme Posteriorgram, derived from Multi-Layer Perceptron trained with a corpus of mixed languages, as features to bring supervised knowledge into the unsupervised task. We also use Hierarchical Agglomerative Clustering algorithm to explore sub-segmental variation of phoneme segments for distinguishing EPs. We tested K-means (assuming known number of EPs) and Gaussian Mixture Model with minimum description length principle (estimating unknown number of EPs) for EP discovery. Preliminary experimental results illustrated the effectiveness of the proposed framework, although there is still a long way to go compared to human annotators.

This paper introduces a new recursive dialogue game frame-work for personalized computer-assisted language learning. A series of sub-dialogue trees are cascaded into a loop as the script for the game. At each dialogue turn there are a number of train-ing sentences to be selected. The dialogue policy is optimized to offer the most appropriate training sentence for an individ-ual learner at each dialogue turn considering the learning status, such that the learner can have the scores for all pronunciation units exceeding a pre-defined threshold in minimum number of turns. The policy is modeled as a Markov Decision Process (MDP) with high dimensional continuous state space. Experi-ments demonstrate promising results for the approach.

Gabor features have been proposed for extracting spectro-temporal modulation information from speech signals, and have been shown to yield large improvements in recognition accuracy. We use a flexible Tandem system framework that integrates multi-stream information including Gabor, MFCC, and pitch features in various ways, by modeling either or both of the tone and phoneme variations in Mandarin speech recognition. We use either phonemes or tonal phonemes (tonemes) as either the target classes of MLP posterior estimation and/or the acoustic units of HMM recognition. The experiments yield a comprehensive analysis on the contributions to recognition accuracy made by either of the feature sets. We discuss their complementarities in tone, phoneme, and toneme classification. We show that Gabor features are better for recognition of vowels and unvoiced consonants, while MFCCs are better for voiced consonants. Also, Gabor features are capable of capturing changes in signals across time and frequency bands caused by Mandarin tone patterns, while pitch features further offer extra tonal information. This explains why the integration of Gabor, MFCC, and pitch features offers such significant improvements.

Error pattern detection is very helpful in Computer-Aided Pronunciation Training (CAPT). This paper reports the work of modeling and detecting Error Patterns defined by language teachers based on their linguist knowledge and pedagogical experiences. We develop a model generation framework to create the Error Pattern models from existing phoneme models. We also propose a serial structure for integrating Goodness-of-Pronunciation with the Error Pattern detectors. Experimental results and analysis over different approaches for modeling and detecting Error Patterns are presented, and it is found that both the binary classification error rates and the capability of Error Pattern diagnosis can be improved effectively with the proposed approaches.