This study explores the potential application of Large Language Models (LLMs) in the Mispronunciation Detection and Diagnosis (MDD) system, which includes pronunciation error detection, feedback, and diagnosis. Accurate detection of incorrect pronunciation, along with comprehensive and effective diagnosis, is key to guiding learners in corrective exercises. Traditional MDD research requires collecting data for specific tasks and training models similar to those used in speech recognition. Moreover, most previous research focuses on identifying the types of errors rather than providing specific pronunciation guidance, resulting in textual feedback on pronunciation errors being relatively limited and the content not sufficiently rich. The recent breakthroughs in LLMs have created new opportunities for pronunciation learning through their ability to generate fluent, educationally valuable feedback, such as explaining error types, demonstrating correct pronunciation, and providing personalized practice guidance. This study explores the potential of multimodal speech models in end-to-end pronunciation error detection and feedback generation. Our experiment results show that comprehensive fine-tuning of the Whisper model using second language (L2) speech data can improve its ability to model L2 speech, thereby increasing the accuracy of mispronunciation detection. The feedback text generated by this model is comparable in quality to the current state-of-the-art (SOTA) level based on LLMs (G-Score of 0.52, compared to SOTA’s 0.54). In addition, this study proposes a pronunciation error feedback method based on pronunciation attribute features using LLMs. The LLMs effectively improve the accuracy and effectiveness of feedback text by analyzing the pronunciation attribute features of incorrect phoneme positions. The evaluation of feedback from LLMs by L2 learners indicates a significant improvement in comprehensibility and helpfulness when using these pronunciation representations. These results confirm the potential of articulatory feature engineering and strategic model optimization in CAPT systems by enhancing learner engagement while reducing instructor workload. • Leveraging Large Language Models to Refine Automatic Feedback Generation. • Fine-tuning multimodal LLM and adding Linguistic knowledge to enhance the mispronunciation detection and diagnosis system. • Substituting edit distance with the Viterbi algorithm for precise phoneme-level error localization between predicted and reference sequences. • Erroneous phonemes are mapped to articulatory attribute features, enabling detailed feedback mechanisms.

The majority of the current mispronunciation detection and diagnosis (MD&D) methods rely on manually annotated data for model training. However, annotating mispronunciations produced by second language (L2) learners is costly. Consequently, data scarcity emerges as a significant challenge in MD&D tasks. In this paper, we employ model-agnostic meta-learning (MAML) to train a phoneme recognition model for MD&D. We conduct experiments using varied meta-learning task partitioning and training strategies to endow the model’s ability to rapidly adapt to unfamiliar speakers. Our best-performing method achieves an F-measure of 61.45%, surpassing both the method using fine-tuned pre-trained model wav2vec2.0 and the approach of incorporating reference text during training. These related works also aim to address the challenge of data scarcity in MD&D. Notably, with few-shot fine-tuning, our model still yielded some remarkable results on F-measure, which suggest that in MD&D tasks, meta-learning is indeed effective.

Voice conversion(VC) model aims to convert the source timbre to the target one. Recently, many VC models utilize pre-trained models to enhance the performance and achieve good results. However, pre-trained models could not somehow disentangle the timbre and linguistic information, thus resulting in a redundancy, which may hurt the conversion performance. In this paper we proposed LIMI-VC, reducing the redundancy between the linguistic content and the timbre information with mutual information disentanglement. We design the model in a light weight form, for the sake of parameter and computation efficiency when pre-trained models are commonly used nowadays. Experiments show that the proposed model can still improve the performance, with 15 times smaller size, compared to baseline. An out-of-domain cross-lingual inference also shows that our model greatly outperforms the baseline. Our source code and audio examples will be available at: https://github.com/WongLaw/LIMI-VC.

Traditionally, the performance of non-native mispronunciation verification systems relied on effective phone-level labelling of non-native corpora. In this study, a multi-view approach is proposed to incorporate discriminative feature representations which requires less annotation for non-native mispronunciation verification of Mandarin. Here, models are jointly learned to embed acoustic sequence and multi-source information for speech attributes and bottleneck features. Bidirectional LSTM embedding models with contrastive losses are used to map acoustic sequences and multi-source information into fixed-dimensional embeddings. The distance between acoustic embeddings is taken as the similarity between phones. Accordingly, examples of mispronounced phones are expected to have a small similarity score with their canonical pronunciations. The approach shows improvement over GOP-based approach by +11.23% and single-view approach by +1.47% in diagnostic accuracy for a mispronunciation verification task.

Formant tracking is one of the most fundamental problems in speech processing.Traditionally, formants are estimated using signal processing methods.Recent studies showed that generic convolutional architectures can outperform recurrent networks on temporal tasks such as speech synthesis and machine translation.In this paper, we explored the use of Temporal Convolutional Network (TCN) for formant tracking.In addition to the conventional implementation, we modified the architecture from three aspects.First, we turned off the "causal" mode of dilated convolution, making the dilated convolution see the future speech frames.Second, each hidden layer reused the output information from all the previous layers through dense connection.Third, we also adopted a gating mechanism to alleviate the problem of gradient disappearance by selectively forgetting unimportant information.The model was validated on the open access formant database VTR.The experiment showed that our proposed model was easy to converge and achieved an overall mean absolute percent error (MAPE) of 8.2% on speech-labeled frames, compared to three competitive baselines of 9.4% (LSTM), 9.1% (Bi-LSTM) and 8.9% (TCN).

L2 learners of Mandarin have difficulty learning native-like pronunciation of nasal codas. In order to help them learn native-like pronunciation, we propose to develop targeted classifiers for automatic pronunciation error detection. In this paper, perceptual experiments with modified speech are designed to analyze the exact position of the landmark of a nasal coda. Based on perceptual results from isolated words, we propose that information about nasal coda place of articulation is most dense near a landmark at the center of the nasalized vowel. Landmarks detected in a database of Japanese learners of Mandarin, and classified as correct vs. incorrect using an SVM. The result shows that the detection performance of the SVM+Landmark system is similar to that of a DNN-HMM+MFCC system. When the two systems are combined, an FRR of 4.6% is achieved at DA of 83.9%. This performance is comparable to that of previously developed classifiers for 16 common Mandarin pronunciation errors.

A segmentation posterior probability based endpointing algorithm for robust ASR is proposed. First, each speech signal is partitioned into homogeneous segments via auto-segmentation. Then posterior probabilities of all possible endpoints are computed, based on the segmentation likelihoods of all levels in a selected range. Endpoints with the highest posterior probabilities are finally selected. The new method differs from the previous auto-segmentation and clustering based algorithm on that the former considers hypotheses from several levels, while the latter depends only on one appropriate level. Another potential benefit of the proposed method is that any endpointing or VAD results can be integrated, as hypotheses, into the posterior probability framework. Experiments based on the AURORA2 digit database show the robustness of the proposed method.

This paper combines Gaussian Mixture Model-Universal Background Model (GMM-UBM) and Support Vector Machine (SVM) through post processing the GMM-UBM scores of different dimension feature parameter with SVM in speaker verification. Because different dimension feature makes different contribution to recognition performance and SVM has good discriminability, this combining approach yields significant performance improvements on decision-making. Experiments on text-independent speaker verification in NIST05 8conv4w-1 conv4w data showed that the actual detection cost function (DCF) of the test system was reduced to 0.0290 from 0.0343.

The acoustic mismatch between the training and test environments will lead to the difference of the statistical characteristics of speech parameters. Since the statistical characteristics of the kurtosis can measure the non-Gaussianity of a random variable, kurtosis normalization will make the training and test speech parameters match the standard normal distribution in some sense. In this paper, a kurtosis normalization method using sigmoid functions (logit functions) in feature space is presented for GMM-UBM based text-independent speaker verification system. Experimental results on the 2004 NIST SRE database show that with the new method significant improvement can be achieved in not only equal error rate but also minimum detection cost compared with baseline system (more than 33% relative reduction for long speech).