21st Annual Conference of the International Speech Communication Association (INTERSPEECH 2020) : Cognitive Intelligence for Speech Processing (Held online due to COVID-19), Shanghai, China, 25 – 29 October 2020

2017-2018 > Academic research: refereed > Refereed conference paper

2017-2018 > Academic research: refereed > Refereed conference paper

Various three-dimensional (3-D) talking heads have been developed lately for language learning, with both external and internal articulatory movements being visualized to guide learning. Mandarin pronunciation animation is challenging due to its confusable stops and affricates with similar places of articulation. Until now, less attention has been paid to the biosignal information of aspiratory airflow, which is essential in distinguishing Mandarin consonants. This study fills a research gap by presenting the quantitative analyses of airflow, and then designing an airflow model for a 3-D pronunciation system. The airflow information was collected by Phonatory Aerodynamic System, so that confusable consonants in Mandarin could be discerned by mean airflow rate, peak airflow rate, airflow duration, and peak time. Based on the airflow parameters, an airflow model using the physical equation of fluid flow was proposed and solved, which was then combined and synchronized with the existing 3-D articulatory model. Therefore, the new multimodal system was implemented to synchronously exhibit the airflow motions and articulatory movements of uttering Mandarin syllables. Both an audio-visual perception test and a pronunciation training study were conducted to assess the effectiveness of our system. Perceptual results indicated that identification accuracy was improved for both native and nonnative groups with the help of airflow motions, while native perceivers exhibited higher accuracy due to long-term language experience. Moreover, our system helped Japanese learners of Mandarin enhance their production skills of Mandarin aspirated consonants, reflected by higher gain values of voice onset time after training.

The 3D talking head has been developed fast, in which both external and internal articulators were demonstrated. For Mandarin pronunciation, the aspiration airflow is crucial to discriminate confusable Mandarin consonants. In this paper, we present a 3D talking head system for articulatory and aspiration animation with the use of EMA articulation data and airflow data simultaneously. The quantitative analyses of airflow data indicated confusable Mandarin consonants could be distinguished from each other by the mean airflow during voicing, peak expiratory airflow, and airflow duration. An airflow model was then incorporated into the 3D articulatory model to produce the airflow in accordance with articulator movements of Mandarin pronunciation. An audio-visual test was designed to evaluate the current 3D articulation and aspiration system, where minimal pairs were used to recognize the animation. The identification accuracy was significantly improved from 43.9% without airflow to 84.8% with airflow-incorporated information.

17th Annual Conference of the International Speech Communication Association, INTERSPEECH 2016, San Francisco, US, 8-16 September 2016

17th Annual Conference of the International Speech Communication Association, INTERSPEECH 2016, San Francisco, US, 8-16 September 2016

17th Annual Conference of the International Speech Communication Association, INTERSPEECH 2016, San Francisco, US, 8-16 September 2016

Multi-path routing and forwarding in non-cooperative networks is extremely challenging due to the co-existence of both rational and Byzantine nodes. They both might deviate from the protocol; however, their intentions and behaviors are totally different. Rational nodes aim to maximize their utilities, while Byzantine nodes purposefully deviate from the protocol to disrupt the normal operation of a network. Most work in the literature treat both kinds of misbehavior without distinction and thus lead to ineffective solutions. This paper presents a hybrid design that seamlessly integrates mechanisms for different misbehavior in a unified framework. The GSP auction provides incentives for rational nodes to cooperate and results in truth-telling Nash equilibria. With the possible inclusion of Byzantine nodes in the least cost paths selected by GSP, the FORBID mechanism builds a decentralized reputation system such that malicious behavior is effectively detected. This in turn triggers the GSP auction to update the least cost paths so as to exclude the malicious nodes from being selected for communication. It is proved that the unified protocol is cooperation-optimal. Experiments have been conducted to further investigate the performance of the proposed protocol and the impact of various parameters.

A visual speller is a brain–computer interface that empowers users with limited motor functionality to input text into a computer by measuring their electroencephalographic responses to visual stimuli. Most prior research on visual spellers has focused on input of alphabetic text. Adapting a speller for other types of segmental or syllabic script is straightforward because such scripts comprise sufficiently few characters that they may all be displayed to the user simultaneously. Logographic scripts, such as Chinese hanzi, however, impose a challenge: How should the thousands of Chinese characters be displayed to the user? Here, we present a visual speller, based on Farwell and Donchin's P300 Speller, for Chinese character input. The speller uses a novel shape-based method called the First–Last, or FLAST, method to encode more than 7,000 Chinese characters. Characters are input by selecting two components, from a set of 56 distinct components, that match the shape of the target character, followed by selection of the character itself. At the input speed of one character per 107 s, 24 able-bodied participants achieved mean online accuracy of 82.8% per component selection and 63.5% per character input. At the faster input speed of one character per 77 s, mean online accuracy was 59.4% per component selection and 33.3% per character input.

Consider multi-path routing and forwarding scenarios in wireless ad hoc networks. Rational and Byzantine nodes both might deviate from the protocol. However, their intentions and behaviors are different. To extend our previous work on generalized second price (GSP) auction for stimulating rational nodes for cooperation, we propose FORBID mechanism in this work to cope with Byzantine behaviors. The core of the FORBID mechanism is a decentralized reputation system. Based on available evidence, each node relies on Bayesian inference to internally update its reputation beliefs that how reliable each other node is. Different from the passive overhearing techniques such as ``watchdog'', each source node under FORBID actively triggers detection process to collect evidence towards Byzantine behaviors. In addition, FORBID includes a flocking algorithm to allow careful dissemination of reputation information and thus shortens the misbehavior detection time.

We model the multi-path routing with selfish nodes as an auction and provide a novel solution from the game-theoretical perspective. By adapting the idea of generalized second price (GSP) payment originating from Internet advertising business and developing pertinent policies for multi-hop networks, we design a mechanism that results in Nash equilibria rather than the traditional strategyproofness, which alleviates the over-payment problem of the widely used Vickrey-Clark-Groves (VCG) payment mechanism. We first provide rigorous theoretical analysis of the proposed mechanism, showing the equilibrium behavior and bounds of the over-payment alleviation, and then evaluate the effectiveness of this protocol through extensive simulations.

We describe a system for highly accurate large-vocabulary Mandarin speech recognition. The prevailing hidden Markov model based technologies are essentially language independent and constitute the backbone of our system. These include minimum-phone-error discriminative training and maximum-likelihood linear regression adaptation, among others. Additionally, careful considerations are taken into account for Mandarin-specific issues including lexical word segmentation, tone modeling, phone set design, and automatic acoustic segmentation. Our system comprises two sets of acoustic models for the purposes of cross adaptation. The systems are designed to be complementary in terms of errors but with similar overall accuracy by using different phone sets and different combinations of discriminative learning. The outputs of the two subsystems are then rescored by an adapted n-gram language model. Final confusion network combination yielded 9.1% character error rate on the DARPA GALE 2007 official evaluation, the best Mandarin recognition system in that year.

We describe a highly accurate large-vocabulary continuous Mandarin speech recognizer, a collaborative effort among four research organizations. Particularly, we build two acoustic models (AMs) with significant differences but similar accuracy for the purposes of cross adaptation and system combination. This paper elaborates on the main differences between the two systems, where one recognizer incorporates a discriminatively trained feature while the other utilizes a discriminative feature transformation. Additionally we present an improved acoustic segmentation algorithm and topic-based language model (LM) adaptation. Coupled with increased acoustic training data, we reduced the character error rate (CER) of the DARPA GALE 2006 evaluation set to 15.3% from 18.4%.

We describe our continuing efforts to improve the UW-SRI-ICSI Mandarin broadcast speech recognizer. This includes increasing acoustic and text training data, adding discriminative features, incorporating frame-level discriminative training criterion, multiplepass acoustic model (AM) cross adaptation, language model (LM) genre adaptation and system combination. The net effect without LM adaptation was a 24%-64 % relative reduction in character error rates (CERs) on a variety of test sets. In addition, LM adaptation gave us another 6 % of relative CER reduction on broadcast conversations. Index Terms — Mandarin, character error rates, discriminative features, discriminative training, LM adaptation. 1.

This paper investigates the issue of automatic segmentation of speech recordings for broadcast news (BN) and broadcast conversation (BC) speech recognition. Our previous segmentation algorithm often exhibited high deletion errors, where some speech segments were misclassified as non-speech and thus were never passed on to the recognizer. In contrast with our previous segmentation models, which only differentiated between speech and non-speech segments, phonetic knowledge is applied to represent speech by using multiple models for different types of speech segments. Moreover, the “pronunciation ” of the speech segment has been modified to loosen the minimum duration constraint. This method makes use of language specific knowledge, while keeping the number of models low to achieve fast segmentation. Experimental results show that the new segmenter outperforms our previous segmenter significantly, particularly in reducing deletion errors. Index Terms: speech recognition, automatic acoustic segmentation, machine translation, broadcast news, broadcast conversation 1.

Tone is an essential component for word formation in all tone languages. Substantial work has been done on using tone information to improve speech recognition of tone languages. In this paper, a new method, called Parallel Tone Score Association (PTSA), for effectively and efficiently using tone in speech recognition is proposed. Experimental results show that the relative character error rates are reduced by as much as 20.94% for Cantonese, and 20.49% for Mandarin compared with the recognition results without tone information. This relative reduction in error rates compares favorably with results reported for other recognition experiments on tone languages.