A complete emotional expression typically contains a complex temporal course in a natural conversation. Related research on utterance-level and segment-level processing lacks understanding of the underlying structure of emotional speech. In this study, a hierarchical affective structure of an emotional utterance characterized by the probabilistic context free grammars (PCFGs) is proposed for emotion modeling. SVM-based emotion profiles are obtained and employed to segment the utterance into emotionally consistent segments. Vector quantization is applied to convert the emotion profile of each segment into codewords. A binary tree in which each node represents a codeword is constructed to characterize the affective structure of the utterance modeled by PCFG. Given an input utterance, the output emotion is determined according to the PCFG-based emotion model with the highest likelihood of the speech segments along with the score of the affective structure. For evaluation, the EMO-DB database and its expansion in utterance length were conducted. Experimental results show that the proposed method achieved emotion recognition accuracy of 87.22% for long utterances and outperformed the SVM-based method.

Based on the clinic data, 69 % - 80 % stroke patients in Taiwan have the muscle weakness symptom of upper limb. Stroke patients are able to restore their independent living if they keep formal rehabilitation without interruption. However, due to the tedious and painful process, stroke patients may pause or suspend their rehabilitation therapy. Moreover, physiatrists in Taiwan hospitals must take care of more than one patient at the same time. This paper is motivated to propose a motor assessment system for stroke upper-limb rehabilitation. Our implementation is integrated with the Complete Minnesota Dexterity Test to quantify patients' recovery status automatically. Therefore, our implementation is able to reduce the workload of physiatrists effectively. In addition to assessment scale score, the proposed system is able to record more details, e.g., the tremor status and the grip strength, for doctors to evaluate the following rehabilitation therapy. Finally, a body-sensed game is provided to increase interest to the rehabilitation process.

This work proposes a novel approach to translate Chinese to Taiwanese sign language and to synthesize sign videos. An aligned bilingual corpus of Chinese and Taiwanese Sign Language (TSL) with linguistic and signing information is also presented for sign language translation. A two-pass alignment in syntax level and phrase level is developed to obtain the optimal alignment between Chinese sentences and Taiwanese sign sequences. For sign video synthesis, a scoring function is presented to develop motion transition-balanced sign videos with rich combinations of intersign transitions. Finally, the maximum a posteriori (MAP) algorithm is employed for sign video synthesis based on joint optimization of two-pass word alignment and intersign epenthesis generation. Several experiments are conducted in an educational environment to evaluate the performance on the comprehension of sign expression. The proposed approach outperforms the IBM Model 2 in sign language translation. Moreover, deaf students perceived sign videos generated by the proposed method to be satisfactory.

This article presents a transfer-based statistical model for Chinese to Taiwanese sign-language (TSL) translation. Two sets of probabilistic context-free grammars (PCFGs) are derived from a Chinese Treebank and a bilingual parallel corpus. In this approach, a three-stage translation model is proposed. First, the input Chinese sentence is parsed into possible phrase structure trees (PSTs) based on the Chinese PCFGs. Second, the Chinese PSTs are then transferred into TSL PSTs according to the transfer probabilities between the context-free grammar (CFG) rules of Chinese and TSL derived from the bilingual parallel corpus. Finally, the TSL PSTs are used to generate the possible translation results. The Viterbi algorithm is adopted to obtain the best translation result via the three-stage translation. For evaluation, three objective evaluation metrics including AER, Top-N, and BLUE and one subjective evaluation metric using MOS were used. Experimental results show that the proposed approach outperforms the IBM Model 3 in the task of Chinese to sign-language translation.

This paper proposes an approach to segmenting and identifying mixed-language speech. A delta Bayesian information criterion (delta-BIC) is firstly applied to segment the input speech utterance into a sequence of language-dependent segments using acoustic features. A VQ-based bi-gram model is used to characterize the acoustic-phonetic dynamics of two consecutive codewords in a language. Accordingly the language-specific acoustic-phonetic property of sequence of phones was integrated in the identification process. A Gaussian mixture model (GMM) is used to model codeword occurrence vectors orthonormally transformed using latent semantic analysis (LSA) for each language-dependent segment. A filtering method is used to smooth the hypothesized language sequence and thus eliminate noise-like components of the detected language sequence generated by the maximum likelihood estimation. Finally, a dynamic programming method is used to determine globally the language boundaries. Experimental results show that for Mandarin, English, and Taiwanese, a recall rate of 0.87 for language boundary segmentation was obtained. Based on this recall rate, the proposed approach achieved language identification accuracies of 92.1% and 74.9% for single-language and mixed-language speech, respectively.

The paper proposes a maximum a posteriori (MAP) based approach to segment and identify jointly an utterance with mixed languages. A statistical framework for language boundary detection and language identification is proposed. First, the MAP estimation is used to determine the boundary number and positions. Further, an LSA-based GMM and a VQ-based bigram language model are proposed to characterize a language and used for language identification. Finally, a likelihood ratio test approach is used to determine the optimal number of language boundaries. Experimental results show that the proposed approach exhibits encouraging potential in mixed-language segmentation and identification.

This paper proposes an efficient error-tolerant approach to retrieving sign words from a Taiwanese Sign Language (TSL) database. This database is tagged with visual gesture features and organized as a multilist code tree. These features are defined in terms of the visual characteristics of sign gestures by which they are indexed for sign retrieval and displayed using an anthropomorphic interface. The maximum a posteriori estimation is exploited to retrieve the most likely sign word given the input feature sequence. An error-tolerant mechanism based on mutual information criterion is proposed to retrieve a sign word of interest efficiently and robustly. A user-friendly anthropomorphic interface is also developed to assist learning TSL. Several experiments were performed in an educational environment to investigate the system's retrieval accuracy. Our proposed approach outperformed a dynamic programming algorithm in its task and shows tolerance to user input errors.

This paper proposes a perceptual modeling approach with a two-stage recognition to deal with the issues of recognition degradation in noisy environment. The auditory masking effect is used for speech enhancement and acoustic modeling in order to overcome the model inconsistencies between training speech and noisy input. In the two-stage recognition, the maximum a posteriori (MAP) based adaptation algorithm is used to incrementally adapt the noise model. In order to evaluate our proposed approach, a Mandarin keyword spotting system was constructed. The experimental results show our proposed method achieves a better recognition rate compared to the audible noise suppression (ANS) and parallel model combination (PMC) methods for both in 70km/hr (10.3dB) and 90km/hr (6.4dB) car environments.