Modeling emotions is an important part of text-to-speech (TTS) research since its goal is to develop a technology for synthesizing naturally sounding speech. In this study, we aimed to build an end-to-end TTS system for the Japanese language that can synthesize emotional speech taking the FastSpeech2 as a base model. Most of the existing approaches use some form of conditioning the generation process on the emotion class label through a corresponding embedding. What separates such methods is the way this conditioning is implemented. Our approach is to add two new blocks which are essentially transformer multi-head self-attention blocks having an input combined with emotion embedding. They are inserted before and after FastSpeech2's variance adaptor and share parameters to ensure robust emotion conditioning when the amount of training data is small. The proposed model was objectively and subjectively evaluated using Mel-Cepstrum Distortion (MCD) and Mean Opinion Score (MOS) criteria respectively. The obtained results show that our approach performs better than a collection of emotion-specific models obtained by fine-tuning the base FastSpeech2.

Recent advances in deep learning based language models have boosted the performance in many downstream tasks such as sentiment analysis, text summarization, question answering, etc. Personality prediction from text is a relatively new task that has attracted researchers' attention due to the increased interest in personalized services as well as the availability of social media data.In this study, we propose a personality prediction system where text embeddings from large language models such as BERT are combined with multiple statistical features extracted from the input text.For the combination, we use the selfattention mechanism which is a popular choice when several information sources need to be merged together.Our experiments with the Kaggle dataset for MBTI clearly show that adding text statistical features improves the system performance relative to using only BERT embeddings.We also analyze the influence of the personality type words on the overall results.

Many studies have shown that articulatory features can significantly improve the performance of automatic speech recognition systems. Unfortunately, such features are not available at recognition time. There are two main approaches to solve this problem: a feature-based approach, the most popular example of which is the acoustic-to-articulatory inversion, where the missing articulatory features are generated from the speech signal, and a model-based approach, where articulatory information is embedded in the model structure and parameters in a way that allows recognition using only acoustic features. In this paper, we propose two new methods to integrate articulatory information into a phoneme recognition system. One of them is feature based, and the other is model based. In both cases, the underlying acoustic model (AM) is a deep neural networks-hidden Markov model (DNN-HMM) hybrid. In the feature-based method, the articulatory inversion DNN and the acoustic model DNN are trained jointly using a linear combination of their loss functions. In the model-based method, we utilize the generalized distillation framework to train the AM DNN. In this case, first, a teacher DNN is trained on both the acoustic and articulatory features, and then its outputs are used as additional targets during the AM DNN training with acoustic features only. A 7-fold cross-validation experiments using 42 speakers from the XRMB database showed that both the proposed methods provide about 22% to 25% performance improvement with respect to the DNN acoustic model trained with acoustic features only.

In the last decade, cheaper and more powerful computations have favored a sufficient surge in research and development of applications in the fields of machine and deep learning. Though often varying in approach, these activities aim mostly at solving similar tasks such as speech synthesis, emotion detection, image recognition, mathematical computations etc. Usually, the typical scenario of using designed algorithms/applications includes inputting data represented in some predefined formats and launching a corresponding program tool that produces expected output data as well as analyzing obtained results. These applications are numerous, and are created by different research teams and laboratories using different techniques and environments including locations, interfaces and procedures to access, query and use them. In a collaborative working environment, developing independent user interfaces for each deep learning application could entail a lot of additional development efforts. In the presented paper, we propose a standardized and flexible interface to reduce design efforts, based on integration of various Deep Learning services (DL services). We also demonstrate a protocol for communication between the user interface and the heterogeneous services. This platform will enable developers of deep learning models to be concerned solely with developing and tuning their models, which can be easily plugged into the central user interface, conveniently exposing their services to users who will have homogeneous central access to a wide-range of DL services, from a unified interface.

Music as a form of art is intentionally composed to be emotionally expressive. The emotional features of music are invaluable for music indexing and recommendation. In this paper we present a cross-comparison of automatic emotional analysis of music. We created a public dataset of Creative Commons licensed songs. Using valence and arousal model, the songs were annotated both in terms of the emotions that were expressed by the whole excerpt and dynamically with 1 Hz temporal resolution. Each song received 10 annotations on Amazon Mechanical Turk and the annotations were averaged to form a ground truth. Four different systems from three teams and the organizers were employed to tackle this problem in an open challenge. We compare their performances and discuss the best practices. While the effect of a larger feature set was not very apparent in the static emotion estimation, the combination of a comprehensive feature set and a recurrent neural network that models temporal dependencies has largely outperformed the other proposed methods for dynamic music emotion estimation.

Availability of large amounts of raw unlabeled data has sparked the recent surge in semi-supervised learning research. In most works, however, it is assumed that labeled and unlabeled data come from the same distribution. This restriction is removed in the self-taught learning approach where unlabeled data can be different, but nevertheless have similar structure. First, a representation is learned from the unlabeled data via sparse coding and then it is applied to the labeled data used for classification. In this work, we implemented this method for the music genre classification task using two different databases: one as unlabeled data pool and the other for supervised classifier training. Music pieces come from 10 and 6 genres for each database respectively, while only one genre is common for both of them. Results from wide variety of experimental settings show that the self-taught learning method improves the classification rate when the amount of labeled data is small and, more interestingly, that consistent improvement can be achieved for a wide range of unlabeled data sizes.

The aim of the present study is to investigate some key challenges of the audio-visual speech recognition technology, such as asynchrony modeling of multimodal speech, estimation of auditory and visual speech significance, as well as stream weight optimization. Our research shows that the use of viseme-dependent significance weights improves the performance of state asynchronous CHMM-based speech recognizer. In addition, for a state synchronous MSHMMbased recognizer, fewer errors can be achieved using stationary time delays of visual data with respect to the corresponding audio signal. Evaluation experiments showed that individual audio-visual stream weights for each visemephoneme pair lead to relative reduction of WER by 20%. Index Terms: multimodal speech, audio-visual processing, Hidden Markov Models, asynchrony, significance weights

In this paper, we describe new language identification system based on the recently developed dynamic hidden Markov network (DHMnet). The DHMnet is a never-ending learning system and provides high resolution model of the speech space. Speech patterns are represented by paths through the network, and these paths when properly labeled with language IDs provide efficient means to discriminate between languages. First experiments indicated that our system can work on-line and is able to deliver relatively high performance with low latency. Evaluated on three language (English, Japanese and Chinese) identification task, the system achieved identification rates of 87.3% and 89.3% for 3 and 5 seconds long speech segments respectively.

Detection of speakers which have not been seen before is an essential part of every online speaker diarization system. New speaker detection accuracy has direct impact on the overall diarization performance. In our previous system, for novelty detection we used global GMM likelihood ratio (LR) threshold. However, as the system analysis showed, the optimal threshold depends on the speaker gender as well as on the number of registered speakers. In this paper, we present the results of this analysis and the approach we have taken to solve this problem. First, we use different thresholds for male and female speakers, and second, for each gender before the thresholding we apply likelihood ratio mean and variance normalization. This greatly reduced the threshold dependency on the number of speakers and allowed to use fixed threshold for each gender. The LR distribution statistics are collected online and updated every time new likelihood ratio is calculated. Experiments on the TCSTAR database showed that compared with the previous global threshold method, the new novelty detection approach reduces the speaker diarization error rate up to 35%.

In this paper, we describe newhigh-performanceon-line speaker diarization system which works faster than real-time and has very low latency. It consists of several modules including voice activity detection, novel speaker detection, speaker gender and speaker identity classification. Allmodules share a set of Gaussian mixturemodels (GMM) representing pause, male and female speakers, and each individual speaker. Initially, there are only three GMMs for pause and two speaker genders, trained in advance from some data. During the speaker diarization process, for each speech segment it is decidedwhether it comes from a new speaker or from already known speaker. In case of a new speaker, his/her gender is identified, and then, from the corresponding gender GMM, a new GMM is spawned by copying its parameters. This GMM is learned on-line using the speech segment data and from this point it is used to represent the new speaker. All individual speaker models are produced in this way. In the case of an old speaker, s/he is identified and the correspondingGMMis again learned on-line. In order to prevent an unlimited grow of the speaker model number, those models that have not been selected as winners for a long period of time are deleted from the system. This allows the system to be able to perform its task indefinitely in addition to being capable of self-organization, i.e. unsupervised adaptive learning, and preservation of the learned knowledge, i.e. speakers. Such functionalities are attributed to the so called Never-Ending Learning systems. For evaluation, we used part of the TC-STAR database consisting of European Parliament Plenary speeches. The results show that this system achieves a speaker diarization error rate of 4.6% with latency of at most 3 seconds.

Current automatic speech recognition systems have two distinctive modes of operation: training and recognition. After the training, system parameters are fixed, and if a mismatch between training and testing conditions occurs, an adaptation procedure is commonly applied. However, the adaptation methods change the system parameters in such a way that previously learned knowledge is irrecoverably destroyed. In searching for a solution to this problem and motivated by the results of recent neuro-biological studies, we have developed a network of hidden Markov states that is capable of unsupervised on-line adaptive learning while preserving the previously acquired knowledge. Speech patterns are represented by state sequences or paths through the network. The network can detect previously unseen patterns, and if such a new pattern is encountered, it is learned by adding new states and transitions to the network. Paths and states corresponding to spurious events or ”noises” and, therefore, rarely visited, are gradually removed. Thus, the network can grow and shrink when needed, i.e. it dynamically changes its structure. The learning process continues as long as the network lasts, i.e. theoretically forever, so it is called neverending learning. The output of the network is the best state sequence and the decoding is done concurrently with the learning. Thus the network always operates in a single learning/decoding mode. Initial experiments with a small database of isolated spelled letters showed that the Dynamic Hidden Markov network is indeed capable of never-ending learning and can perfectly recognize previously learned speech patterns. Index Terms: never-ending learning, life-long learning, dynamic hidden markov network, self-organization, topology representation.

We introduce a method of incorporating additional knowledge sources into an HMM-based statistical acoustic model. The probabilistic relationship between information sources is first learned through a Bayesian network to easily integrate any ad-ditional knowledge sources that might come from any domain and then the global joint probability density function (PDF) of the model is formulated. Where the model becomes too com-plex and direct BN inference is intractable, we utilize a junction tree algorithm to decompose the global joint PDF into a linked set of local conditional PDFs. This way, a simplified form of the model can be constructed and reliably estimated using a limited amount of training data. Here, we apply this framework to in-corporate accents, gender, and wide-phonetic knowledge infor-mation at the HMM phonetic model level. The performance of the proposed method was evaluated on an LVCSR task using two different types of accented English speech data. Experi-mental results revealed that our method improves word accu-racy with respect to standard HMM. Index Terms: acoustic model, knowledge incorporation, Bayesian network, junction tree algorithm, wide-phonetic knowledge. 1.

This paper introduces a general framework for incorporating additional sources of knowledge into an HMM-based statistical acoustic model. Since the knowledge sources are often derived from different domains, it may be difficult to formulate a probabilistic function of the model without learning the causal dependencies between the sources. We utilized a Bayesian network framework to solve this problem. The advantages of this graphical model framework are 1) it allows the probabilistic relationship between information sources to be learned and 2) it facilitates the decomposition of the joint probability density function (PDF) into a linked set of local conditional PDFs. This way, a simplified form of the model can be constructed and reliably estimated using a limited amount of training data. We applied this framework to the problem of incorporating wide-phonetic knowledge information, which often suffers from a sparsity of data and memory constraints. We evaluated how well the proposed method performed on an large-vocabulary continuous speech recognition (LVCSR) task using English speech data that contained two different types of accents. The experimental results revealed that it improved the word accuracy with respect to standard HMM, with or without additional sources of knowledge.

This paper presents a new method of modeling pentaphone-context units using the hybrid HMM/BN acoustic modeling. Rather than modeling pentaphones explicitly, in this approach we extend the modeled phonetic context within the triphone framework, since the probabilistic dependencies between the triphone context unit and the second preceding/following contexts are incorporated into the triphone state output distributions by means of the BN. Another advantage is that we can use a standard decoding system by assuming the next preceding/following context variables hidden during recognition. In this study, the performance of pentaphone HMM/BN model was evaluated with our LVCSR system by phoneme recognition and by large-vocabulary continuous word recognition tasks. In both cases, we observed consistently improved performance over the standard HMM based triphone model with the same number of parameters

In this paper, we describe an application of the Forward-Backwards (F-B) algorithm for maximum likelihood training of hybrid HMM/Bayesian Network (BN) acoustic models. Previously, HMM/BN parameter estimation was based on a Viterbi training algorithm that requires two passes over the training data: one for BN learning and one for updating HMM transition probabilities. In this work, we first analyze the F-B training for a conventional HMM and show that the state PDF parameter estimation is analogous to weighted-data classifier training. The gamma variable of the Forward-Backwards algorithm plays the role of the data weight. From this perspective, it is straightforward to apply FB-based training to the HMM/BN models since the BN learning algorithm allows training with weighted data. Experiments on accented speech (American, British and Australian English) show that F-B training outperforms the previous Viterbi learning approach and that the HMM/BN model achieved better performance than the conventional HMM. Index Terms: forward-backwards algorithm, HMM/BN, weighted data training, accent modeling.

We propose a new method of incorporating the additional knowledge of accent, gender, and wide-context dependency information into ASR systems by utilizing the advantages of Bayesian networks. First, we only incorporate pentaphone-context dependency information. After that, accent and gender information are also integrated. In this method, we can easily extend conventional triphone HMMs to cover various sources of knowledge. The probabilistic dependencies between a triphone context unit and additional knowledge are learned through a BN. Another advantage is that during recognition, additional knowledge variables are assumed to be hidden, so that the existing standard triphone-based decoding system can be used without modification. The performance of the proposed model was evaluated on an LVCSR task using two different types of accented English speech data. Experimental results show that this proposed method improves word accuracy with respect to standard triphone models. Index Terms: acoustic modeling, bayesian network, knowledge incorporation, wide-context dependency.

In this paper, we describe the ATR multilingual speech-to-speech translation (S2ST) system, which is mainly focused on translation between English and Asian languages (Japanese and Chinese). There are three main modules of our S2ST system: large-vocabulary continuous speech recognition, machine text-to-text (T2T) translation, and text-to-speech synthesis. All of them are multilingual and are designed using state-of-the-art technologies developed at ATR. A corpus-based statistical machine learning framework forms the basis of our system design. We use a parallel multilingual database consisting of over 600 000 sentences that cover a broad range of travel-related conversations. Recent evaluation of the overall system showed that speech-to-speech translation quality is high, being at the level of a person having a Test of English for International Communication (TOEIC) score of 750 out of the perfect score of 990.

Most current state-of-the-art speech recognition systems use the hidden Markov model (HMM) for modeling the acoustical characteristics of a speech signal. In the first-order HMM, speech data are assumed to be independently and identically distributed (iid), meaning that there is no dependency between neighboring feature vectors. Another assumption is that the current vector depends only on the current HMM state. In practice, however, these assumptions are not true. We describe a hybrid HMM/BN (Bayesian network) acoustic model, where the dependency of the current speech vector on the previous vector and on the previous state is also learned and used in speech recognition. This is possible because the state probability distribution is modeled by a BN. Previous instances of the state and speech feature vector are represented by additional variables of the BN and the probabilistic dependencies between them, and their current instances are learned during training. During recognition, the likelihood of the current feature vector is inferred from the BN where the previous state and previous feature vector are treated as hidden. We have evaluated this hybrid HMM/BN model with our LVCSR system by phoneme recognition and by large-vocabulary continuous word recognition tasks. In both cases, we observed improved performance over the conventional Gaussian mixture HMM.

This paper presents a method for improving acoustic model precision by incorporating wide phonetic context units in speech recognition. The wide phonetic context model is constructed from several narrower context-dependent models based on the Bayesian framework. Such a composition is performed in order to avoid the crucial problem of a limited availability of training data and to reduce the model complexity. To enhance the model reliability due to unseen contexts and limited training data, flooring and deleted interpolation techniques are used. Experimental results show that this method gives improvement of the word accuracy with respect to the standard triphone model.

In this paper, we describe several approaches to integra-tion of the articulatory dynamic parameters along with ar-ticulatory position data into a HMM/BN model based au-tomatic speech recognition system. This work is a contin-uation of our previous study, where we have successfully combined speech acoustic features in form of MFCC with articulatory position observations. Articulatory dynamic parameters are represented by velocity and acceleration coefficients calculated as first and second derivatives of the articulatory position data. All these features are in-tegrated using the HMM/BN acoustic model where each feature corresponds to different Bayesian Network vari-able. By changing the BN topology we can change the way articulatory and acoustic parameters are combined. The evaluation experiments showed that the effect of the articulatory dynamic features greatly depends on the BN structure and that careful data analysis is essential in gain-ing knowledge about the underlying dependencies be-tween different information sources. In comparison with conventional HMM system trained on acoustic data only, the HMM/BN system achieved significant improvement of the recognition performance. 1.

It is difficult to recognize speech distorted by various factors, especially when an ASR system contains only a single acoustic model. One solution is to use multiple acoustic models, one model for each different condition. In this paper, we discuss a parallel decoding-based ASR system that is robust to the noise type, SNR, speaker gender and speaking style. Our system consists of two recognition channels based on MFCC and Differential MFCC (DMFCC) features. Each channel has several acoustic models depending on SNR, speaker gender and speaking style, and each acoustic model is adapted by fast noise adaptation. From each channel, one hypothesis is selected based on its likelihood. The final recognition result is obtained by combining hypotheses from the two channels. We evaluate the performance of our system by normal and hyperarticulated test speech data contaminated by various types of noise at different SNR levels. Experiments demonstrate that the system could achieve recognition accuracy in excess of 80% for the normal speaking style data at a SNR of 0 dB. For hyper-articulated speech data, the recognition accuracy improved from about 10% to over 45% compared to a system without acoustic models for hyperarticulated speech.

In current HMM based speech recognition systems, it is difficult to supplement acoustic spectrum features with additional information such as pitch, gender, articulator positions, etc. On the other hand, dynamic Bayesian networks (DBN) allow for easy combination of different features and make use of conditional dependencies between them. However, lack of efficient algorithms has prevented their application in large vocabulary continuous speech recognition. The hybrid HMM/BN acoustic model, where HMM are used for modeling of temporal speech characteristics and state probability model is represented by BN, provides a trade off solution to the problem. In this paper we describe the HMM/BN acoustic model and LVCSR system built upon this model. In the HMM/BN model, in addition to speech observation variable, state BN has two more discrete variables representing speaker gender and pitch frequency. Evaluation results on WSJ database showed lower word error rate with respect to the same complexity conventional HMM acoustic model when there is enough training data to estimate reliable HMM/BN parameters.

In this paper, we describe automatic speech recognition system where features extracted from human speech production system in form of articulatory movements data are effectively integrated in the acoustic model for improved recognition performance. The system is based on the hybrid HMM/BN model, which allows for easy integration of different speech features by modeling probabilistic dependencies between them. In addition, features like articulatory movements, which are difficult or impossible to obtain during recognition, can be left hidden, in fact eliminating the need of their extraction. The system was evaluated in phoneme recognition task on small database consisting of three speakers’ data in speaker dependent and multi-speaker modes. In both cases, we obtained higher recognition rates compared to conventional, spectrum based HMM system with the same number of parameters.

We present the maximum normalized likelihood estimation (MNLE) algorithm and its application for discriminative training of hidden Markov models (HMMs) for continuous speech recognition. The objective of this algorithm is to maximize the normalized frame likelihood of training data. Instead of gradient descent techniques usually applied for objective function optimization in other discriminative algorithms such as the minimum classification error (MCE) and maximum mutual information (MMI), we used a modified expectation-maximization (EM) algorithm which greatly simplifies and speeds up the training procedure. Evaluation experiments showed better recognition rates compared, to both the maximum likelihood (ML) training method and MCE/GPD discriminative method. In addition, the MNLE algorithm showed better generalization abilities and was faster than MCE/GPD.

In most of the current speech recognition systems based on HMM, existing decoding and utterance veri cation methods make use of state output likelihood as a measure of the acoustic match between the input data and the acoustic models. In this paper, we present a new and more generalized approach to the formation of the acoustic match score. The essence of this approach is to transform the likelihood of each acoustic vector with respect to any particular HMM state according to some non-linear function. We have investigated two types of such transformation functions. The rst one, performs likelihood normalization, and the second one transforms likelihoods into exponentially ordered weights. The transformed likelihoods, as new acoustic scores, are used further for decoding, recognition and veri cation instead of the conventional likelihoods. In our evaluation experiments we used TIMIT database for phoneme recognition and veri cation and a database of 710 speakers and a total of 4252 distinct words, for isolated word recognition and veri cation. The results we achieved show that the transformed likelihood scores, in average, increase slightly the recognition accuracy and reduce the veri cation error rates up to 30%.

We propose a new method, where the likelihood normalization technique is applied at both the frame and utterance levels. In this method based on Gaussian mixture models (GMM), every frame of the test utterance is inputed to the claimed and all background speaker models in parallel. In this procedure, for each frame, likelihoods from all the background models are available, hence they can be used for normalization of the claimed speaker likelihood at every frame. A special kind of likelihood normalization, called weighting models rank, is also proposed. We have evaluated our method using two databases-TIMIT and NTT. Results show that the combination of frame and utterance level likelihood normalization in some cases reduces the equal error rate (EER) more than twice.

In this paper we propose a new speaker identication system, where the likelihood normalization technique, widely used for speaker verication, is introduced.In the new system, which is based on Gaussian Mixture Models, every frame of the test utterance is inputed to all the reference models in parallel.In this procedure, for each frame, likelihoods from all the models are available, hence they can be normalized at every frame.A special kind of likelihood normalization, called Weighting Models Rank, is also proposed.Experiments were performed using two databases -TIMIT and NTT.Evaluation results clearly show that frame level likelihood normalization technique is superior to the standard accumulated likelihood approach.