Recent advancements in speech synthesis have significantly improved the audio quality and pronunciation of synthesized speech.To further advance toward human-like conversational speech synthesis, this paper presents Filler-Speech, a novel speech synthesis framework that enables natural filler insertion and control over filler style.To address this, we construct a filler-inclusive speech data, derived from the open-source large-scale speech corpus.This data includes fillers with pitch and duration information.For the generation and style control of natural fillers, we propose a method that tokenizes the filler style and utilizes crossattention with the input text.Furthermore, we introduce a large language model-based filler prediction method that enables natural insertion of fillers even when only text input is provided.The experimental results demonstrate that the constructed dataset is valid and that our proposed methods for filler style control and filler prediction are effective.

Recently, there has been a growing demand for conversational speech synthesis (CSS) that generates more natural speech by considering the conversational context. To address this, we introduce JELLY, a novel CSS framework that integrates emotion recognition and context reasoning for generating appropriate speech in conversation by fine-tuning a large language model (LLM) with multiple partial LoRA modules. We propose an Emotion-aware Q-former encoder, which enables the LLM to perceive emotions in speech. The encoder is trained to align speech emotions with text, utilizing datasets of emotional speech. The entire model is then fine-tuned with conversational speech data to infer emotional context for generating emotionally appropriate speech in conversation. Our experimental results demonstrate that JELLY excels in emotional context modeling, synthesizing speech that naturally aligns with conversation, while mitigating the scarcity of emotional conversational speech datasets.

Controlling singing style is crucial for achieving an expressive and natural singing voice. Among the various style factors, vibrato plays a key role in conveying emotions and enhancing musical depth. However, modeling vibrato remains challenging due to its dynamic nature, making it difficult to control in singing voice conversion. To address this, we propose VibESVC, a controllable singing voice conversion model that explicitly extracts and manipulates vibrato using discrete wavelet transform. Unlike previous methods that model vibrato implicitly, our approach decomposes the F0 contour into frequency components, enabling precise transfer. This allows vibrato control for enhanced flexibility. Experimental results show that VibE-SVC effectively transforms singing styles while preserving speaker similarity. Both subjective and objective evaluations confirm high-quality conversion.

Emotional text-to-speech (TTS) has advanced significantly, but challenges persist due to the complexity of emotions and limitations in emotional speech datasets and models. A key issue with previous studies is the reliance on limited emotional speech datasets or extensive manual annotations, which restrict generalization across different speakers and emotional styles. To address this, we propose EmoSphere++, an emotioncontrollable zero-shot TTS model capable of generating expressive speech with fine-grained control over emotional style and intensity—without requiring manual annotations. We introduce a novel emotion-adaptive spherical vector that effectively captures emotional style and intensity, along with a joint attribute style encoder that enhances generalization to both seen and unseen speakers. To further improve emotion transfer in zero-shot scenarios, we introduce an additional disentanglement method to enhance the style transfer performance for zero-shot scenarios. Through both objective and subjective evaluations, we demonstrate the benefits of the proposed model in emotion style and intensity modeling, as well as its effectiveness in enhancing emotional expressiveness across both seen and unseen speakers

Emotional voice conversion (EVC) involves modifying various acoustic characteristics, such as pitch and spectral envelope, to match a desired emotional state while preserving the speaker’s identity. Existing EVC methods often rely on text transcriptions or time-alignment information and struggle to handle varying speech durations effectively. In this paper, we propose DurFlex-EVC, a duration-flexible EVC framework that operates without the need for text or alignment information. We introduce a unit aligner that models contextual information by aligning speech with discrete units representing content, eliminating the need for text or speech-text alignment. Additionally, we design a style autoencoder that effectively disentangles content and emotional style, allowing precise manipulation of the emotional characteristics of the speech. We further enhance emotional expressiveness through a hierarchical stylize encoder that applies the target emotional style at multiple hierarchical levels, refining the stylization process to improve the naturalness and expressiveness of the converted speech. Experimental results from subjective and objective evaluations demonstrate that our approach outperforms baseline models, effectively handling duration variability and enhancing emotional expressiveness in the converted speech.

Expressive text-to-speech systems have undergone significant advancements owing to prosody modeling, but conventional methods can still be improved. Traditional approaches have relied on the autoregressive method to predict the quantized prosody vector; however, it suffers from the issues of long-term dependency and slow inference. This study proposes a novel approach called DiffProsody in which expressive speech is synthesized using a diffusion-based latent prosody generator and prosody conditional adversarial training. Our findings confirm the effectiveness of our prosody generator in generating a prosody vector. Furthermore, our prosody conditional discriminator significantly improves the quality of the generated speech by accurately emulating prosody. We use denoising diffusion generative adversarial networks to improve the prosody generation speed. Consequently, DiffProsody is capable of generating prosody 16 times faster than the conventional diffusion model. The superior performance of our proposed method has been demonstrated via experiments.