The voice-based Internet of Multimedia Things (IoMT) is the combination of IoT interfaces and protocols with associated voice-related information, which enables advanced applications based on human-to-device interactions. An example is Automatic Speech Recognition (ASR) for live captioning and voice translation. Three major issues of ASR for IoMT are IoT development cost, speech recognition accuracy, and execution time complexity. For the first issue, most non-voice IoT applications are upgraded with the ASR feature through hard coding, which are error prone. For the second issue, recognition accuracy must be improved for ASR. For the third issue, many multimedia IoT services are real-time applications and, therefore, the ASR delay must be short. This article elaborates on the above issues based on an IoT platform called VoiceTalk. We built the largest Taiwanese spoken corpus to train VoiceTalk ASR (VT-ASR) and show how the VT-ASR mechanism can be transparently integrated with existing IoT applications. We consider two performance measures for VoiceTalk: speech recognition accuracy and VT-ASR delay. For the acoustic tests of PAL-Labs, VT-ASR's accuracy is 96.47%, while Google's accuracy is 94.28%. We are the first to develop an analytic model to investigate the probability that the VT-ASR delay for the first speaker is complete before the second speaker starts talking. From the measurements and analytic modeling, we show that the VT-ASR delay is short enough to result in a very good user experience. Our solution has won several important government and commercial TV contracts in Taiwan. VT-ASR has demonstrated better Taiwanese Mandarin speech recognition accuracy than famous commercial products (including Google and Iflytek) in Formosa Speech Recognition Challenge 2018 (FSR-2018) and was the best among all participating ASR systems for Taiwanese recognition accuracy in FSR-2020.

Before the era of the neural network (NN), features extracted from auditory models have been applied to various speech applications and been demonstrated more robust against noise than conventional speech-processing features. What's the role of auditory models in the current NN era? Are they obsolete? To answer this question, we construct a NN with a generative auditory model embedded to process speech signals. The generative auditory model consists of two stages, the stage of spectrum estimation in the logarithmic-frequency axis by the cochlea and the stage of spectral-temporal analysis in the modulation domain by the auditory cortex. The NN is evaluated in a simple speaker identification task. Experiment results show that the auditory model embedded NN is still more robust against noise, especially in low SNR conditions, than the randomly-initialized NN in speaker identification.

A new data-driven approach to building a speaking rate-dependent hierarchical prosodic model (SR-HPM), directly from a large prosody-unlabeled speech database containing utterances of various speaking rates, to describe the influences of speaking rate on Mandarin speech prosody is proposed. It is an extended version of the existing HPM model which contains 12 sub-models to describe various relationships of prosodic-acoustic features of speech signal, linguistic features of the associated text, and prosodic tags representing the prosodic structure of speech. Two main modifications are suggested. One is designing proper normalization functions from the statistics of the whole database to compensate the influences of speaking rate on all prosodic-acoustic features. Another is modifying the HPM training to let its parameters be speaking-rate dependent. Experimental results on a large Mandarin read speech corpus showed that the parameters of the SR-HPM together with these feature normalization functions interpreted the effects of speaking rate on Mandarin speech prosody very well. An application of the SR-HPM to design and implement a speaking rate-controlled Mandarin TTS system is demonstrated. The system can generate natural synthetic speech for any given speaking rate in a wide range of 3.4-6.8 syllables/sec. Two subjective tests, MOS and preference test, were conducted to compare the proposed system with the popular HTS system. The MOS scores of the proposed system were in the range of 3.58-3.83 for eight different speaking rates, while they were in 3.09-3.43 for HTS. Besides, the proposed system had higher preference scores (49.8%-79.6%) than those (9.8%-30.7%) of HTS. This confirmed the effectiveness of the speaking rate control method of the proposed TTS system.

This paper proposes a maximum intelligibility (MI)-based close-loop speech synthesis framework to actively compensate for the distortion of background noises. In this framework, an extra environmental noise-sensing microphone and an automatic speech recognition (ASR) module are utilized to approximate a subjective intelligibility measure. The hidden Markov model-based speech synthesis system (HTS) is then online adjusted by using the MI-based model adaptation algorithm. Experimental results of two subjective listening tests in noisy environments show that the proposed approach obtains 64% of the votes in an A/B preference test and helps the participants reduce word dictation errors by relative 26% when compared to an HTS baseline.

In this paper, a subband minimum classification error beamforming (S-MCEBEAM), instead of the subband likelihood maximizing beamforming (S-LIMABEAM) proposed by Seltzer, is investigated to closely integrate microphone array and speech recognizer for robust speech recognition in reverberant environments. The main idea behind this is to apply minimum classification error (MCE) criterion to directly match the goal of automatic speech recognition (ASR) and to simultaneously adjust both beamformer parameters and recognizer's acoustic models. Experimental results on a Mandarin reverberation corpus created from Mandarin spontaneous speech corpus (TCC300) and RWCP's sound scene database show S-MCEBEAM leads to better recognition results than S-LIMABEAM in reverberant environments.

In continuous speech recognition substitution, insertion and deletion errors usually not only vary in numbers but also have different degrees of impact on optimizing a set of acoustic models. To balance their contributions to the overall error, an enhanced minimum classification error (E-MCE) learning framework is developed. The basic idea is to partition acoustic model optimization into three subtasks, i.e., minimum substitution errors (MSE), insertion errors (MIE) and deletion errors (MDE), and select/generate three corresponding sets of competing hypotheses, one for each individual sub-problem. MSE, MIE and MDE are then sequentially executed to gradually reduce the overall word error rates. Experimental results on continuous Mandarin digit recognition of five different data sets collected over various acoustic conditions have consistently shown the effectiveness of the proposed E-MCE learning framework.

The major difficulty of prosody modeling and automatic tone recognition of continuous Mandarin speech is the complex interaction of tones and prosody/intonation on FO contours. In this study, we propose a latent prosody model (LPM) aiming to jointly model the affections of tone and prosody state on FO. The main purposes are twofold including (1) automatic prosody state labeling and (2) improving tone recognition accuracy. The basic idea is to introduce latent prosody state variables into an additive statistic model of FO which already considers the affecting factors of tone and speaker. Experiments on the Tree-Bank corpus showed that LPM not only gave meaningful prosody state labeling results but also improved the average tone recognition rate from 80.86% of a multi-layer perceptron (MLP) baseline to 82.55%.

Handsets that are not seen in the training phase (unseen handsets) are significant sources of performance degradation for speaker identification (SID) applications in the telecommunication environment. In this paper, a novel latent prosody analysis (LPA) approach to automatically extract the most discriminative prosodic cues for assisting in conventional spectral feature-based SID is proposed. The concept of the LPA approach is to transform the SID problem into a full-text document retrieval-like task via 1) prosodic contour tokenization, 2) latent prosody analysis, and 3) speaker retrieval. Experimental results of the phonetically balanced, read-speech, handset-TIMIT (HTIMIT) database demonstrated that the proposed method of fusing the LPA prosodic feature-based SID systems with maximum-likelihood a priori handset knowledge interpolation (ML-AKI) spectral feature-based SID outperformed both the pitch and energy Gaussian mixture model (Pitch-GMM) and the bigram of the prosodic state (Bigram) counterparts for both cases of counting all and only unseen handsets.

In this investigation, two probabilistic latent semantic analyses (PLSA)-based approaches are proposed for use in speaker verification systems to reduce the number of parameters required by prosodic speaker models to (1) estimate reliably speakers' bi-gram models and to (2) reduce the amount of required training and test data. The basic concept is to (1) adopt PLSA to smooth the underlying n-gram-based prosodic speaker models, and to (2) use PLSA to find a compact latent prosody space to represent efficiently the constellation of speakers. The proposed approaches are evaluated on the standard single-speaker detection task of the 2001 NIST Speaker Recognition Evaluation Corpus, where only one 2 minute training enrollment speech and 30 s test speech on average are available. Experimental results demonstrated that the proposed approach can reduce the required number of bi-gram parameters from 112 to 88 and 63 per speaker and improve the EERs of MAP-GMM and GMM+T-norm from 12.4% and 9.5% to 10.4% and 8.4%, respectively, and finally to 8.1% after fusing all systems

Unseen handset mismatch and limited training/test data are the major source of performance degradation for speaker identification in telecommunication environments. In this paper, a vector quantization (VQ)-based prosody modeling and an eigen-prosody analysis (EPA) is integrated to transform the close-set speaker identification problem into a full text document retrieval-similar task. The prosody modeling labels the prosodic feature contours of a speaker's speech into sequences of prosody states. EPA then constructs a compact eigen-prosody space to represent the constellation of speakers. Furthermore, EPA is fused with a lower-level a priori knowledge interpolation (AKI) handset distortion compensator to complement each other. Experimental results on the HTIMIT database had shown that about 41.0% and 32.8% relative error rate reduction for seen and unseen handsets, respectively, was achieved compared with the maximum a priori-adapted Gaussian mixture model/cepstral mean subtraction (MAP-GMM/CMS) baseline.

A novel eigen-prosody analysis approach is proposed for robust speaker recognition under a mismatch handset environment. The idea is to convert the prosodic contours of a speaker's speech into sequences of prosody symbols, and transform the speaker recognition problem into a full-text document retrieval-similar task. Experimental results on the HTIMIT corpus have shown that, even though only few training/test data are available, about 32.2% relative error rate reduction could be achieved compared with the conventional Gaussian mixture model/cepstral mean subtraction approach.

Unseen but mismatch handset is the major source of performance degradation for speaker recognition in telecommunication environment. In this paper, an unseen handset characteristics estimation method based on a priori knowledge interpolation (AKI) is proposed. AKI could be applied in both the feature and model space to interpolate the feature and model transformation functions measured using stochastic matching (SM) and maximum likelihood linear regression (MLLR), respectively. Cross-validation experimental results on HTIMIT database showed that the average speaker recognition rate could be improved from 59.6%/57.8 % to 73.8%/66.8 % for seen/unseen handsets. It is therefore a promising method for robust speaker recognition. 1.

We present a mismatch-aware stochastic matching (MASM) algorithm to alleviate the performance degradation under mismatched training and testing conditions. MASM first computes a reliability measure of applying a set of pre-trained speech models to a mismatch test utterance along the time axis or among different feature vector components. It then estimates and compensates the mismatch using the reliability measure to guide the speech segmentation. Experiments on a serious mismatched condition with training on PSTN-speech database and testing on mobile GSM-speech database showed that MASM outperformed the stochastic match (SM) method, especially, for short utterances.

In this paper, an iterative bandwidth mismatch compensation (BMC) algorithm is proposed to alleviate the need of multiple pre-trained models for recognizing different bandwidth speech. The BMC uses the concept of the bandwidth extension as similar as in the speech enhancement approaches. However, it aims at directly improving the recognition accuracy instead of speech intelligence or quality and utilizes only recognizer’s hidden Markov models (HMMs) for both bandwidth mismatch compensation and recognition. The BMC first detects the bandwidth of the input speech signal based on a divergence measurement. The HMM/Gaussian mixture model (GMM)based method is then used to iteratively segment the input speech utterance and compensates the speech features. Experiments on serious bandwidth mismatched conditions, i.e., training on 8 kHz and testing on 4 kHz or 5.5 kHz bandwidth database have verified the effectiveness of the proposed approach.

The issue of this paper is to enable continuous speech input for directory assistance tasks. We are focusing on handling of long lists of Chinese names. Compared with the more restrictive isolated word input scenario the name error rate for the continuous task is much larger. By concentrating on the name segment, we investigate two methods in order to bridge the performance gap. First, a confidence measure-based rescoring approach to pickup the most confident name is presented. Then, a segmentation-based two-pass method, first estimating the time segmentation of the name and secondly, applying the isolated word recognition technique is studied. The obtained results for both methods reveal a significant name recognition error rate reduction by up to 42% relatively.

A new modular recurrent neural network (MRNN)-based method for continuous Mandarin speech recognition (CMSR) is proposed. The MRNN recognizer is composed of four main modules. The first is a sub-MRNN module whose function is to generate discriminant functions for all 412 base-syllables. It accomplishes the task by using four recurrent neural network (RNN) submodules. The second is an RNN module which is designed to detect syllable boundaries for providing timing cues in order to help solve the time-alignment problem. The third is also an RNN module whose function is to generate discriminant functions for 143 intersyllable diphone-like units to compensate the intersyllable coarticulation effect. The fourth is a dynamic programming (DP)-based recognition search module. Its function is to integrate the other three modules and solve the time-alignment problem for generating the recognized base-syllable sequence. A new multilevel pruning scheme designed to speed up the recognition process is also proposed. The whole MRNN can be trained by a sophisticated three-stage minimum classification error/generalized probabilistic descent (MCE/GPD) algorithm. Experimental results showed that the proposed method performed better than the maximum likelihood (ML)-trained hidden Markov model (HMM) method and is comparable to the MCE/GPD-trained HMM method. The multilevel pruning scheme was also found to be very efficient.

In this paper, we present the Philips large vocabulary continuous Mandarin speech recognition system developed for the 2000 Taiwan Speech Input Technology Assessment. We systematically integrated key Mandarin components with up-todate Western-language techniques to build up a state-of-the-art Mandarin speech recognition system. These technologies include robust pitch extraction/tone modeling, context-dependent preme/core-final units, Chinese phrase/syllable trigram language model, linear discriminant analysis ( LDA), cross-syllable modeling/decoding, speaker clustering and maximum likelihood linear regression (MLLR) adaptation. Among them, the major breakthroughs were our robust pitch extraction/tone modeling technology and the treatment of coarticulation across syllable boundaries. For the development set, we dramatically reduced last year’s best error rates by relative 44.8%~67.8% on all three categories we participated. Moreover, for the evaluation set, we achieved the lowest unit error rates on all three categories.

A new modular recurrent neural network (MRNN)-based method for continuous Mandarin speech recognition is proposed. The system uses five RNNs to accomplish many subtasks separately and then combine them to integrally solve the problem. They include two RNNs for the discrimination of the two sub-syllable groups of 100 right-final-dependent (RFD) initials and 39 context independent (CI) finals, two RNNs for the generation of dynamic weighting functions for sub-syllable's integration, and one RNN for syllable boundary detection. All RNN modules are combined using a delay-decision Viterbi search. The method differs from the ANN/HMM hybrid approach of using ANNs to perform not only sub-syllables discrimination but also temporal structure modeling of the speech signal. The system is trained using a three-stage training method embedding with the MCE/GPD algorithms. Besides, a fast recognition method using multi-level pruning is also proposed. Experimental results showed that it outperforms the HMM method on both the recognition accuracy and the computational complexity.

A novel recurrent neural network-based (RNN-based) front-end preclassification scheme for fast continuous Mandarin speech recognition is proposed. First, an RNN is employed to discriminate each input frame for the three broad classes of initial, final, and silence. A finite state machine (FSM) is then used to classify the input frame into four states including three stable states of initial (I), final (F), and silence (S), and a transient (T) state. The decision is made based on examining whether the RNN discriminates well between classes. We then restrict the search space for the three stable states in the following DP search to speed up the recognition process. The efficiency of the proposed scheme was examined by simulations in which we incorporate it with a hidden Markov model-based (HMM-based) continuous 411 Mandarin based-syllables recognizer. The experimental results showed that it can be used in conjunction with the beam search to greatly reduce the computational complexity of the HMM recognizer while keeping the recognition rate almost undegraded.

A new modular recurrent neural network (MRNN)- based speech-recognition method that can recognize the entire vocabulary of 1280 highly confusable Mandarin syllables is proposed in this paper. The basic idea is to first split the complicated task, in both feature and temporal domains, into several much simpler subtasks involving subsyllable and tone discrimination, and then to use two weighting RNN's to generate several dynamic weighting functions to integrate the subsolutions into a complete solution. The novelty of the proposed method lies mainly in the use of appropriate a priori linguistic knowledge of simple initial-final structures of Mandarin syllables in the architecture design of the MRNN. The resulting MRNN is therefore effective and efficient in discriminating among highly confusable Mandarin syllables. Thus both the time-alignment and scaling problems of the ANN-based approach for large-vocabulary speech-recognition can be addressed. Experimental results show that the proposed method and its extensions, the reverse-time MRNN (Rev-MRNN) and bidirection MRNN (Bi-MRNN), all outperform an advanced HMM method trained with the MCE/GPD algorithm in both recognition-rate and system complexity.

An ANN-based continuous Mandarin base-syllable recognition system is proposed. It adopts a hybrid approach to combine an HRNN with a Viterbi search. The HRNN is taken at a front-end processor and responsible for calculating discrimination scores for all 411 base-syllables. The Viterbi search is then followed to find out the best base-syllable sequence with highest score as the recognized output. Experimental results showed that the proposed system outperforms the conventional HMM method on both the recognition accuracy and the computational complexity. The system can also be further modified to reduce the computational complexity while retaining the recognition accuracy almost be ungraded.

A new method is proposed to train recurrent neural networks (RNNs) for speech recognition such that the difficulty of selecting appropriate target functions can be avoided. A novel architecture of the RNN-based speech recognition system is also introduced for solving the problem related to large vocabulary speech recognition. Additionally, the proposed RNN-based recognizer is found to have the advantages of being capable of absorbing the temporal variation of speech patterns as well as possessing effective discrimination capabilities. Performance of the proposed system was examined using two speech recognition tasks of recognizing 10 Mandarin digits and 54 confusable Mandarin syllables. Experimental results show that the proposed method outperforms both the continuous observation densities hidden Markov models method and a RNN recognizer using the extended back propagation training algorithm.>