In recent years, the rapid progress in speaker verification (SV) technology has been driven by the extraction of speaker representations based on deep learning. However, such representations are still vulnerable to emotion variability. To address this issue, we propose multiple improvements to train speaker encoders to increase emotion robustness. Firstly, we utilize CopyPaste-based data augmentation to gather additional parallel data, which includes different emotional expressions from the same speaker. Secondly, we apply cosine similarity loss to restrict parallel sample pairs and minimize intraclass variation of speaker representations to reduce their correlation with emotional information. Finally, we use emotion-aware masking (EM) based on the speech signal energy on the input parallel samples to further strengthen the speaker representation and make it emotion-invariant. We conduct a comprehensive ablation study to demonstrate the effectiveness of these various components. Experimental results show that our proposed method achieves a relative 19.29% drop in EER compared to the baseline system.

Generally, the performance of deep neural networks (DNNs) heavily depends on the quality of data representation learning. Our preliminary work has emphasized the significance of deep representation learning (DRL) in the context of speech enhancement (SE) applications. Specifically, our initial SE algorithm employed a gated recurrent unit variational autoencoder (VAE) with a Gaussian distribution to enhance the performance of certain existing SE systems. Building upon our preliminary framework, this paper introduces a novel approach for SE using deep complex convolutional recurrent networks with a VAE (DCCRN-VAE). DCCRN-VAE assumes that the latent variables of signals follow complex Gaussian distributions that are modeled by DCCRN, as these distributions can better capture the behaviors of complex signals. Additionally, we propose the application of a residual loss in DCCRN-VAE to further improve the quality of the enhanced speech. Compared to our preliminary work, DCCRN-VAE introduces a more sophisticated DCCRN structure and probability distribution for DRL. Furthermore, in comparison to DCCRN, DCCRN-VAE employs a more advanced DRL strategy. The experimental results demonstrate that the proposed SE algorithm outperforms both our preliminary SE framework and the state-of-the-art DCCRN SE method in terms of scale-invariant signal-to-distortion ratio, speech quality, and speech intelligibility.

Recently, researchers have shown an increasing interest in automatically predicting the subjective evaluation for speech synthesis systems. This prediction is a challenging task, especially on the out-of-domain test set. In this paper, we proposed a novel fusion model for MOS prediction that combines both supervised and unsupervised approaches. In the supervised aspect, we developed a SSL-based predictor called LE-SSL-MOS. The LE-SSL-MOS utilizes pre-trained self-supervised learning models and further improves prediction accuracy by utilizing the opinion scores of each utterance in the listener enhancement branch. In the unsupervised aspect, two steps are contained: one is that we fine-tuned unit language model (ULM) using highly-intelligible domain data to improve the correlation of an unsupervised metric SpeechLMScore. Another is that we utilized ASR confidence as a new metric with the help of ensemble learning. To the best of our knowledge, this is the first architecture that fuses supervised and unsupervised methods for MOS prediction.With these approaches, our experimental results on the VoiceMOS Challenge 2023 show that LE-SSL-MOS performs better than the baseline. Our fusion system achieved an absolute improvement of 13 % over LE-SSL-MOS on the noisy and enhanced speech track. And our system ranked 1st and 2 nd respectively in the French speech synthesis track and the noisy and enhanced speech track of the challenge.

Voice User Interfaces (VUIs) are popular among older adults, who find them easy to use and perceive them as social companions. However, there is a lack of research on voice-based applications to support physical activities for older adults. To address this gap, we present "Workout Pal," a voice agent designed to deliver physical activities to older adults. We conducted a mixed-methods study involving ten older adults to understand their perceptions and design priorities when interacting with Workout Pal. Questionnaires and semi-structured interviews were used to assess their experience, while experience-based co-design sessions facilitated collaboration in exploring the design space and identifying design requirements. Our findings highlight the feasibility and design preferences for smart speaker-based physical activity programs. The study contributes by sharing the design and development of Workout Pal, illustrating a novel co-design approach with older adults, and providing design implications tailored to the specific needs of older adults. The design guidelines emphasize the importance of sociability, voice design, and individualization. This research supports the development of elder-friendly VUIs helping older adults live independently and engage in physical activities.

This study explores whether there is a predictive relationship between humans’ ethical perspectives and their trust in autonomous systems (AS). Whether AS can make ethically acceptable decisions, especially in safety-critical situations involving value trade-offs, has become a significant determinant of how humans will trust these systems. However, knowledge about these relation-based trust dimensions is largely absent from the current theoretical framework of human-AS trust. Addressing this gap, this study used MTurk and Qualtrics to perform an online survey that assessed people’s ethical perspectives, trust in automation, and propensity to trust. The results showed that: (1) Significant differences in trust in automation were seen across four ethical perspectives, confirming the predictive relationship between human ethical perspectives and AS trust. (2) There was no significant difference in propensity to trust among ethical orientations. (3) There was a positive but weak association between trust in automation and willingness to trust. The latter two observations jointly show that trust in automation and trust propensity may be regulated by distinct mechanisms. This study has contributed to existing knowledge by (1) validating the predictive relationship between human ethical perspectives and how they trust AS, (2) revealing potential mechanisms underlying such discrepancies, and (3) highlighting how these differences could help the design of trustworthy AS.

This paper describes our RoyalFlush system for the track of multi-speaker automatic speech recognition (ASR) in the M2MeT challenge. We adopted the serialized output training (SOT) based multi-speakers ASR system with large-scale simulation data. Firstly, we investigated a set of front-end methods, including multi-channel weighted predicted error (WPE), beamforming, speech separation, speech enhancement, etc., to process training, evaluation, and test sets. However, according to their experimental results, we only selected the WPE and beamforming approach as our front-end methods. Secondly, we made great efforts in the data augmentation for multi-speaker ASR, including adding noise and reverberation, over-lapped speech simulation, multi-channel speech simulation, speed perturbation, front-end processing, etc., which brought us a significant performance improvement. Finally, to make full use of the performance complementary of different model architecture, we trained the standard conformer based joint CTC/Attention (Conformer) and U2++ ASR model with a bidirectional attention decoder, a modification of Conformer, to fuse their results. Compared with the official baseline system, our system got a 12.22% absolute Character Error Rate (CER) reduction on the evaluation set and 12.11% on the test set.

The acoustic modeling unit is crucial for an end-to-end speech recognition system, especially for the Mandarin language. Until now, most of the studies on Mandarin speech recognition focused on individual units, and few of them paid attention to using a combination of these units. This paper uses a hybrid of the syllable, Chinese character, and subword as the modeling units for the end-to-end speech recognition system based on the CTC/attention multi-task learning. In this approach, the character-subword unit is assigned to train the transformer model in the main task learning stage. In contrast, the syllable unit is assigned to enhance the transformer’s shared encoder in the auxiliary task stage with the Connectionist Temporal Classification (CTC) loss function. The recognition experiments were conducted on AISHELL-1 and an open data set of 1200-hour Mandarin speech corpus collected from the OpenSLR, respectively. The experimental results demonstrated that using the syllable-char-subword hybrid modeling unit can achieve better performances than the conventional units of char-subword, and 6.6% relative CER reduction on our 1200-hour data. The substitution error also achieves a considerable reduction.

Cloud computing heralded an era where resources can be scaled up and down elastically and in an online manner. This paper initiates the study of cost-effective cloud resource allocation algorithms under price discounts, using a competitive analysis approach. We show that for a single resource, the online resource renting problem can be seen as a 2-dimensional variant of the classic online parking permit problem, and we formally introduce the PPP2problem accordingly. Our main contribution is an online algorithm for PPP2which achieves a deterministic competitive ratio of k (under a certain set of assumptions), where k is the number of resource bundles. This is almost optimal, as we also prove a lower bound of k/3 for any deterministic online algorithm. Our online algorithm makes use of an optimal offline algorithm, which may be of independent interest since it is the first optimal offline algorithm for the 1D and 2D versions of the parking permit problem. Finally, we show that our algorithms and results also generalize to multiple resources (i.e., Multi-dimensional parking permit problems).

This paper presents our recent progress on translating TED speeches1, a collection of public lectures covering a variety of topics. Specially, we use word-to-word alignment to compose translation units of bilingual tuples and present a recurrent neural network-based translation model (RNNTM) to capture long-span context during estimating translation probabilities of bilingual tuples. However, this RNNTM has severe data sparsity problem due to large tuple vocabulary and limited training data. Therefore, a factored RNNTM, which takes bilingual tuples in addition to source and target phrases of the tuples as input features, is proposed to partially address the problem. Our experimental results on the IWSLT2012 test sets show that the proposed models significantly improve the translation quality over state-of-the-art phrase-based translation systems.

This study presents an overview of VoiceTra, which was developed by NICT and released as the world's first network-based multilingual speech-to-speech translation system for smartphones, and describes in detail its multilingual speech recognition, its multilingual translation, and its multilingual speech synthesis in regards to field experiments. We show the effects of system updates using the data collected from field experiments to improve our acoustic and language models.

Spoken Document Retrieval (SDR) and Spoken Term Detection (STD) have been two of the most intensively investigated topics in spoken document processing research according to the establishment of the SDR and STD test collections by the Text REtrieval Conference (TREC) and NIST. Because Japanese spoken document processing researchers also requires such test collections for SDR and STD, we have established a working group to develop these collections in Special Interest Group-Spoken Language Processing (SIG-SLP) of the Information Processing Society of Japan. The working group has constructed and made available a test collection for SDR, and is now constructing new test collections for STD that will be open to researchers. The present paper introduces the policies, outline, and schedule of the new test collections. Then, the new test collections are compared with the NIST STD test collections. Index Terms: spoken term detection, test collection 1.

Conventional question answering (QA) techniques independently process candidate-bearing snippets to select an exact answer to a question from candidate answers. This paper presents two novel ways of utilizing redundancy in candidate-bearing snippets to help select an exact answer to a question in our Web QA system, i.e., cluster-based language model (CLM-M) and unsupervised SVM classifier (U-SVM) techniques. The comparative experiments demonstrate that the proposed methods significantly outperform the language model-based (LM-M) and supervised SVM-based (S-SVM) techniques that do not utilize this redundancy in the candidate-bearing snippets. Using the CLM-M, the top_1 score is increased from 36.03% (LM-M) to 46.96%; and the top_1 improvement in the U-SVM over the S-SVM is about 23%. Moreover, a cross-model comparison shows that the performance ranking of these models is: U-SVM > CLM-LM > LM-M > S-SVM > R-M (the retrieval-based model).

Priority scheduling has been widely used in mobile ad hoc networks. However, most of the prior work related to priority scheduling is designed only for a single data channel. In addition, without providing certain mechanisms to incorporate priority scheduling, existing multi-channel MAC protocols can not provide differentiated service. In this paper, we propose a multi-channel priority MAC protocol which can significantly increase the throughput of high priority flows and reduce their average delay. Our protocol consists of four mechanisms: control phase contention mechanism, priority- oriented channel assignment mechanism, data phase contention mechanism and flow interpolation mechanism. Simulation results demonstrate the effectiveness of the proposed protocol.

An appropriate approach to model tone information is helpful for building Chinese large vocabulary continuous speech recognition system. We propose to derive an efficient phoneme set of tone-dependent sub-word units to build a recognition system, by iteratively merging a pair of tone-dependent units according to the principle of minimal loss of the mutual information. The mutual information is measured between the word tokens and their phoneme transcriptions in a training text corpus, based on the system lexical and language model. The approach has the capability to keep discriminative tonal (and phoneme) contrasts that are most helpful for disambiguating homophone words due to lack of tones, and merge those tonal (and phoneme) contrasts that are not important for word disambiguation for the recognition task. This enable a flexible selection of phoneme set according to a balance between the MI information amount and the number of phonemes. We applied the method to traditional phoneme set of Initial/Finals, and derived several phoneme sets with different number of units. Speech recognition experiments using the derived sets showed their effectiveness.