Abstract This paper proposes a sequence-to-sequence model for data-to-text generation, called DM-NLG, to generate a natural language text from structured nonlinguistic input. Specifically, by adding a dynamic memory module to the attention-based sequence-to-sequence model, it can store the information that leads to generate previous output words and use it to generate the next word. In this way, the decoder part of the model is aware of all previous decisions, and as a result, the generation of duplicate words or incomplete semantic concepts is prevented. To improve the generated sentences quality by the DM-NLG decoder, a postprocessing step is performed using the pretrained language models. To prove the effectiveness of the DM-NLG model, we performed experiments on five different datasets and observed that our proposed model is able to reduce the slot error rate rate by 50% and improve the BLEU by 10%, compared to the state-of-the-art models.

Abstract In this paper, we propose an enhanced version of the vanilla transformer for data-to-text generation and then use it as the generator of a conditional generative adversarial model to improve the semantic quality and diversity of output sentences. Specifically, by adding a diagonal mask matrix to the attention scores of the encoder and using the history of the attention weights in the decoder, this enhanced version of the vanilla transformer prevents semantic defects in the output text. Also, using this enhanced transformer along with a triplet network, respectively, as the generator and discriminator of conditional generative adversarial network, diversity and semantic quality of sentences are guaranteed. To prove the effectiveness of the proposed model, called conditional generative adversarial with enhanced transformer (CGA-ET), we performed experiments on three different datasets and observed that our proposed model is able to achieve better results than the baselines models in terms of BLEU, METEOR, NIST, ROUGE-L, CIDEr, BERTScore, and SER automatic evaluation metrics as well as human evaluation.

In this paper, we combine Hidden Markov Models (HMMs) with i-vector extractors to address the problem of text-dependent speaker recognition with random digit strings. We employ digit-specific HMMs to segment the utterances into digits, to perform frame alignment to HMM states and to extract Baum-Welch statistics. By making use of the natural partition of input features into digits, we train digit-specific i-vector extractors on top of each HMM and we extract well-localized i-vectors, each modelling merely the phonetic content corresponding to a single digit. We then examine ways to perform channel and uncertainty compensation, and we propose a novel method for using the uncertainty in the i-vector estimates. The experiments on RSR2015 part III show that the proposed method attains 1.52% and 1.77% Equal Error Rate (EER) for male and female respectively, outperforming state-of-the-art methods such as x-vectors, trained on vast amounts of data. Furthermore, these results are attained by a single system trained entirely on RSR2015, and by a simple score-normalized cosine distance. Moreover, we show that the omission of channel compensation yields only a minor degradation in performance, meaning that the system attains state-of-the-art results even without recordings from multiple handsets per speaker for training or enrolment. Similar conclusions are drawn from our experiments on the RedDots corpus, where the same method is evaluated on phrases. Finally, we report results with bottleneck features and show that further improvement is attained when fusing them with spectral features.

We present our work on improving the numerator graph for discriminative training using the lattice-free maximum mutual information (MMI) criterion. Specifically, we propose a scheme for creating unconstrained numerator graphs by removing time constraints from the baseline numerator graphs. This leads to much smaller graphs and therefore faster preparation of training supervisions. By testing the proposed un-constrained supervisions using factorized time-delay neural network (TDNN) models, we observe 0.5% to 2.6% relative improvement over the state-of-the-art word error rates on various large-vocabulary speech recognition databases.

In recent years, end-to-end approaches to automatic speech recognition have received considerable attention as they are much faster in terms of preparing resources. However, conventional multistage approaches, which rely on a pipeline of training hidden Markov models (HMM)-GMM models and tree-building steps still give the state-of-the-art results on most databases. In this study, we investigate flat-start one-stage training of neural networks using lattice-free maximum mutual information (LF-MMI) objective function with HMM for large vocabulary continuous speech recognition. We thoroughly look into different issues that arise in such a setup and propose a standalone system, which achieves word error rates (WER) comparable with that of the state-of-the-art multi-stage systems while being much faster to prepare. We propose to use full biphones to enable flat-start context-dependent (CD) modeling and show through experiments that our CD modeling approach can be almost as effective as regular tree-based CD modeling. We show that our flat-start LF-MMI setup together with this tree-free CD modeling technique achieves 10 to 25 % relative WER reduction compared to other end-to-end methods on well-known databases. The improvements are larger for smaller databases.

The low-dimensional i-vector representation of speech segments is used in the state-of-the-art text-independent speaker verification systems. However, i-vectors were deemed unsuitable for the text-dependent task, where simpler and older speaker recognition approaches were found more effective. In this work, we propose a straightforward hidden Markov model (HMM) based extension of the i-vector approach, which allows i-vectors to be successfully applied to text-dependent speaker verification. In our approach, the Universal Background Model (UBM) for training phrase-independent i-vector extractor is based on a set of monophone HMMs instead of the standard Gaussian Mixture Model (GMM). To compensate for the channel variability, we propose to precondition i-vectors using a regularized variant of within-class covariance normalization, which can be robustly estimated in a phrase-dependent fashion on the small datasets available for the text-dependent task. The verification scores are cosine similarities between the i-vectors normalized using phrase-dependent s-norm. The experimental results on RSR2015 and RedDots databases confirm the effectiveness of the proposed approach, especially in rejecting test utterances with a wrong phrase. A simple MFCC based i-vector/HMM system performs competitively when compared to very computationally expensive DNN-based approaches or the conventional relevance MAP GMM-UBM, which does not allow for compact speaker representations. To our knowledge, this paper presents the best published results obtained with a single system on both RSR2015 and RedDots dataset.

Abstract In this article we tackle the problem of multi-party conversation summarization. We investigate the role of discourse segmentation of a conversation on meeting summarization. First, an unsupervised function segmentation algorithm is proposed to segment the transcript into functionally coherent parts, such asMonologuei(which indicates a segment where speakeriis the dominant speaker, e.g., lecturing all the other participants) orDiscussionx1x2, . . .,xn(which indicates a segment where speakersx1toxninvolve in a discussion). Then the salience score for a sentence is computed by leveraging the score of the segment containing the sentence. Performance of our proposed segmentation and summarization algorithms is evaluated using the AMI meeting corpus. We show better summarization performance over other state-of-the-art algorithms according to different metrics.

In this paper, we aim to improve meeting summarization performance using discourse specific information. Since there are intrinsically different characteristics in utterances in different types of function segments, e.g., Monologue segments versus Discussion ones, we propose a new summarization framework where different summarizers are used for different segment types. For monologue segments, we adopt the integer linear programming-based summarization method; whereas for discussion segments, we use a graph-based method to incorporate speaker information. Performance of our proposed method is evaluated using the standard AMI meeting corpus. Results show a good improvement over previous state-of-the-art algorithms according to various evaluation metrics and different compress ratios.

I-vectors have proved to be the most effective features for text-independent speaker verification in recent researches. In this article a new scheme is proposed to utilize i-vectors in text-prompted speaker verification in a simple while effective manner. In order to examine this scheme empirically, a telephony dataset of Persian month names is introduced. Experiments show that the proposed scheme reduces the EER by 31% compared to the state-of-the-art State-GMM-MAP method. Furthermore it is shown that using HMM instead of GMM for universal background modeling leads to 15% reduction in EER.

In this paper, statistical speech enhancement using hidden Markov model (HMM) is studied and new techniques for applying non‐Gaussian distributions are proposed. The superiority of using non‐Gaussian distributions in online adaptive noise suppression algorithms has been proven; however, in this study, this approach is formulated in an HMM‐based mean‐square error estimator (MMSE) estimator in which a priori models are trained in an off‐line manner. In addition, an analytical study of using different distributions other than autoregressive (AR) Gaussian distribution, such as Laplace, is presented in order to construct an accurate HMM as a priori model for discrete Fourier transform and discrete cosine transform feature vectors of speech signal. In the proposed framework, an HMM‐based MMSE estimator bassed on Gaussian assumption using diagonal covariance matrix is provided rather than AR hypothesis which is employed in the conventional AR‐HMM‐based speech enhancement algorithm. Experimental evaluations of the proposed methods are done in the presence of four different noise types at various signal‐to‐noise ratio levels which demonstrate the superiority of the proposed methods in most conditions in comparison with AR‐HMM.

Linear segmentation of a meeting conversation is beneficial as a stand-alone system (to organize a meeting and make it easier to access) or as a preprocessing step for many other meeting related tasks. Such segmentation can be done according to two different criteria: topic in which a meeting is segmented according to the different items in its agenda, and function in which the segmentation is done according to the meeting's different events (like discussion, monologue). In this article we concentrate on the function segmentation task and propose new unsupervised methods to segment a meeting into functionally coherent parts. The first proposed method assigns a score to each possible boundary according to its local information and then selects the best ones. The second method uses a dynamic programming approach to find the global best segmentation according to a defined cost function. Since these two methods are complementary of each other, we propose the third method as a combination of the first two ones, which takes advantage of both to improve the final segmentation. In order to evaluate our proposed methods, a subset of a standard meeting dataset (AMI) is manually annotated and used as the test set. Results show that our proposed methods perform significantly better than the previous unsupervised approach according to different evaluation metrics.

A novel distance measure for distance-based speaker segmentation is proposed. This distance measure is nonparametric, in contrast to common distance measures used in speaker segmentation systems, which often assume a Gaussian distribution when measuring the distance between 1 two audio segments. This distance measure is essentially a k-nearestneighbor distance measure. Non-vowel segment removal in preprocessing stage is also proposed. Speaker segmentation performance is tested on artificially created conversations from the TIMIT database and two AMI conversations. For short window lengths, Missed Detection Rated is decreased significantly. For moderate window lengths, a decrease in both Missed Detection and False Alarm Rates occur. The computational cost of the distance measure is high for long window lengths. Index Terms: speaker segmentation, distance measure, k-nearest-neighbor

This paper studies a number of well-known noise estimation techniques and provides a comparative performance analysis of them in speech enhancement platform. Two types of evaluation data that simulate consistent and inconsistent noisy conditions are prepared in the presence of six noise types at different SNR levels. The performance of speech enhancement systems and the spectrum distance of the estimated and original noise spectrums are used as evaluation criteria. The evaluations indicate that a simple VAD method outperforms noise estimation methods in most of the consistent noisy conditions.

A new voice activity detection (VAD) algorithm with soft decision output in Mel-frequency domain is developed based on hidden Markov model (HMM) and is incorporated in an HMM-based speech enhancement system. The proposed VAD uses a two-state ergodic HMM representing speech presence and speech absence. The states are constructed from noisy speech and noise HMMs used in the speech enhancement system. This composite model provides a robust detection of speech segments in the presence of noise and obviates the need for extra modeling in HMM-based speech enhancement applications. As the main purpose of the proposed VAD is to detect speech segments accurately, a hang-over mechanism is proposed and is applied on the output of the VAD to improve the speech detection rate. The VAD is integrated in the HMM-based speech enhancement system in Mel-frequency spectral (MFS) and cepstral (MFC) domains. The performance of the proposed VAD, the effectiveness of the hang-over mechanism and the performance of the VAD-integrated speech enhancement system are evaluated on four noise types at different SNR levels. The experimental results confirm the superiority of the proposed VAD compared to the reference methods particularly for speech detection rate at the dominant noisy conditions.

In this paper, a problem in spoken dialogue systems namely the menu problem, is introduced and solved by a POMDP model. To overcome the large size of the menu problem, a new method for achieving an optimal policy called divided POMDP method is introduced. Conditions for the problem to be solved by the proposed method are specified and the problem properties resulting in the given conditions are presented. The proposed method is evaluated using a typical menu problem with different menu sizes and it is shown that this method is superior to the conventional methods such as FRTDP for the problems it is capable to solve. Moreover, it converges faster in getting to an optimal policy.

In this paper a novel method, called PC-PMC, is proposed to improve the performance of automatic speech recognition systems in noisy environments. This method is based on the parallel model combination (PMC) technique and uses the cepstral mean subtraction (CMS) normalization ability and principal component analysis (PCA) compression and de-correlation capabilities. It takes the advantages of both additive noise compensation of PMC and convolutive noise removal ability of CMS and PCA. The first problem to be solved in the realizing of PC-PMC is that PMC algorithm requires invertible modules in the front-end of the system while CMS normalization is not an invertible process. Also, it is required to design a framework for adaptation of the PCA transform in the presence of noise. The method proposed in this paper provides solutions to the both problems. Our evaluations are done on four different real noisy tasks using Nevisa Persian continuous speech recognition system. Experimental results demonstrate significant reduction in word error rate using PC-PMC in comparison with the standard robustness methods.

Class-based n-gram language models are those most frequently-used in continuous speech recognition systems, especially for languages for which no richly annotated corpora are available. Various word clustering algorithms have been proposed to build such class-based models. In this work, we discuss the superiority of soft approaches to class construction, whereby each word can be assigned to more than one class. We also propose a new method for possibilistic word clustering. The possibilistic C-mean algorithm is used as our clustering method. Various parameters of this algorithm are investigated; e.g., centroid initialization, distance measure, and words' feature vector. In the experiments reported here, this algorithm is applied to the 20,000 most frequent Persian words, and the language model built with the clusters created in this fashion is evaluated based on its perplexity and the accuracy of a continuous speech recognition system. Our results indicate a 10% reduction in perplexity and a 4% reduction in word error rate.

This paper proposes a distant speech recognition system based on a novel speaker localization and beamforming (SRLB) algorithm. To localize the speaker an algorithm based on steered response power by utilizing harmonic structures of speech signal is proposed. This new scheme has the ability of speaker verification by fundamental frequency variation; therefore it can be utilized in the design of a speech recognition system for verified speakers. Then the performance of the Farsi speech recognition engine is evaluated under notorious conditions of noise and reverberation. Simulation results and tests on real data shows that by utilizing proposed localization scheme, recognition accuracy improves by 28% in high noise and reverberant conditions compared to the accuracy of single channel recognition. The capability of this algorithm in localizing a verified speaker improves system robustness to speech noises and reduces recognition errors up to %52 in the presence of speech noise.

An improved hidden Markov model-based (HMM-based) speech enhancement system designed using the minimum mean square error principle is implemented and compared with a conventional spectral subtraction system. The improvements to the system are: (1) incorporation of mixture components in the HMM for noise in order to handle noise nonstationarity in a more flexible manner, (2) two efficient methods in the speech enhancement system design that make the system real-time implementable, and (3) an adaptation method to the noise type in order to accommodate a wide variety of noise expected under the enhancement system's operating environment. The results of the experiments designed to evaluate the performance of the HMM-based speech enhancement systems in comparison with spectral subtraction are reported. Three types of noise-white noise, simulated helicopter noise, and multitalker (cocktail party) noise-were used to corrupt the test speech signals. Both objective (global SNR) and subjective mean opinion score (MOS) evaluations demonstrate consistent superiority of the HMM-based enhancement systems that incorporate the innovations described in this paper over the conventional spectral subtraction method.

Transitional speech units for American English are proposed and constructed via assimilation of active articulatory features. The effectiveness of a feature-based system exploiting the transitional speech units is demonstrated in evaluation experiments, where the new system with use of quadratic regressive states is shown to achieve error rate reduction of 21% compared with the system using only static subphonemic units.

We provide a formal description of a speech recognizer designed on the basis of elaborate articulatory timing that a asynchronous across the multiple articulatory-feature dimensions. Three improved critical components of the recognizer are described in detail. Evaluation results, obtained from a standard TIMIT phonetic recognition task confined within the N-best rescoring scenario, are reported on comparative performances between the new feature-based recognizer and a recognizer using the conventional context-dependent triphone units. The results demonstrate an overall superior quality of the rescored N-best list from the feature-based recognizer over that from the triphone-based recognizer. Greater performance improvements are observed as the top number of candidate sentences increases.

We describe our recent work on the real-time implementation of a state-of-the-art HMM-based MMSE speech enhancement algorithm, where our earlier published algorithm has been approximated, optimized, and simplified. The key innovations enabling the enhancement system to run in real-time are: (1) an algorithm for automatic selection of the noise model, (2) pruning the MMSE forward calculations, (3) a new pause detection method useable for a SNR down to 0 dB, and (4) task partitioning of the entire system. A preliminary version of the real-time enhancement system is simulated on the IBM-PC 486DX2/66 and at the same time implemented on a DSP platform based on dual TMS320C30 DSP chips using single precision floating point arithmetic.

We have investigated effectiveness of several HMM-based speech enhancement strategies in the hearing aid context and compared their performance to a traditional method based on spectral subtraction (also called Wiener filtering). The performance of the various systems has been evaluated using several types of noise likely to be encountered by hearing impaired listeners. Both the objective and subjective evaluations demonstrate consistent superiority of the HMM-based enhancement methods over the spectral subtraction method. The superiority is more evident for sentences contaminated by nonstationary noises at low input SNRs.>