Controlling text-to-speech (TTS) systems to synthesize speech with the prosodic characteristics expected by users has attracted much attention. To achieve controllability, current studies focus on two main directions: (1) using reference speech as prosody prompt to guide speech synthesis, and (2) using natural language descriptions to control the generation process. However, finding reference speech that exactly contains the prosody that users want to synthesize takes a lot of effort. Description-based guidance in TTS systems can only determine the overall prosody, which has difficulty in achieving fine-grained prosody control over the synthesized speech. In this paper, we propose DrawSpeech, a sketch-conditioned diffusion model capable of generating speech based on any prosody sketches drawn by users. Specifically, the prosody sketches are fed to DrawSpeech to provide a rough indication of the expected prosody trends. DrawSpeech then recovers the detailed pitch and energy contours based on the coarse sketches and synthesizes the desired speech. Experimental results show that DrawSpeech can generate speech with a wide variety of prosody and can precisely control the fine-grained prosody in a user-friendly manner. Our implementation and audio samples are publicly available1.

We propose Universal target audio Separation (UniSep), addressing the separation task on arbitrary mixtures of different types of audio. Distinguished from previous studies, UniSep is performed on unlimited source domains and unlimited source numbers. We formulate the separation task as a sequence-to-sequence problem, and a large language model (LLM) is used to model the audio sequence in the discrete latent space, leveraging the power of LLM in handling complex mixture audios with large-scale data. Moreover, a novel pre-training strategy is proposed to utilize audio-only data, which reduces the efforts of large-scale data simulation and enhances the ability of LLMs to understand the consistency and correlation of information within audio sequences. We also demonstrate the effectiveness of scaling datasets in an audio separation task: we use large-scale data (36.5k hours), including speech, music, and sound, to train a universal target audio separation model that is not limited to a specific domain. Experiments show that UniSep achieves competitive subjective and objective evaluation results compared with single-task models.

Speech emotions are crucial in human communication and are extensively used in fields like speech synthesis and natural language understanding. Most prior studies, such as speech emotion recognition, have categorized speech emotions into a fixed set of classes. Yet, emotions expressed in human speech are often complex, and categorizing them into predefined groups can be insufficient to adequately represent speech emotions. On the contrary, describing speech emotions directly by means of natural language may be a more effective approach. Regrettably, there are not many studies available that have focused on this direction. Therefore, this paper proposes a speech emotion captioning framework named SECap, aiming at effectively describing speech emotions using natural language. Owing to the impressive capabilities of large language models in language comprehension and text generation, SECap employs LLaMA as the text decoder to allow the production of coherent speech emotion captions. In addition, SECap leverages HuBERT as the audio encoder to extract general speech features and Q-Former as the Bridge-Net to provide LLaMA with emotion-related speech features. To accomplish this, Q-Former utilizes mutual information learning to disentangle emotion-related speech features and speech contents, while implementing contrastive learning to extract more emotion-related speech features. The results of objective and subjective evaluations demonstrate that: 1) the SECap framework outperforms the HTSAT-BART baseline in all objective evaluations; 2) SECap can generate high-quality speech emotion captions that attain performance on par with human annotators in subjective mean opinion score tests.

Prosodic boundary plays an important role in text-to-speech synthesis (TTS) in terms of naturalness and readability.However, the acquisition of prosodic boundary labels relies on manual annotation, which is costly and time-consuming.In this paper, we propose to automatically extract prosodic boundary labels from text-audio data via a neural text-speech model with pre-trained audio encoders.This model is pre-trained on text and speech data separately and jointly fine-tuned on TTS data in a triplet format: {speech, text, prosody}.The experimental results on both automatic evaluation and human evaluation demonstrate that: 1) the proposed text-speech prosody annotation framework significantly outperforms text-only baselines; 2) the quality of automatic prosodic boundary annotations is comparable to human annotations; 3) TTS systems trained with model-annotated boundaries are slightly better than systems that use manual ones.Code is released 1 .

This paper investigates the application of pre-trained BERT model in three classic text analysis tasks: Chinese grapheme-to-phoneme(G2P), text normalization(TN) and sentence punctuation annotation. Even though the full-sized BERT has prominent modeling power, there are two challenges for it in real applications: the requirement for annotated training data and the considerable computational cost. In this paper, we propose BERT-based low-latency solutions. To collect sufficient training corpus for G2P, we transfer knowledge from existing rule-based system to BERT through a large amount of unlabeled corpus. The new model could convert all characters directly from raw texts with higher accuracy. We also propose a hybrid two-stage text normalization pipeline which reduces the sentence error rate by 25% compared to the rule-based system. We offer both supervised and weakly supervised versions and find that the latter has only 1% accuracy drop from the former.

In spoken conversations, spontaneous behaviors like filled pause and prolongations always happen.Conversational partner tends to align features of their speech with their interlocutor which is known as entrainment.To produce human-like conversations, we propose a unified controllable spontaneous conversational speech synthesis framework to model the above two phenomena.Specifically, we use explicit labels to represent two typical spontaneous behaviors filled-pause and prolongation in the acoustic model and develop a neural network based predictor to predict the occurrences of the two behaviors from text.We subsequently develop an algorithm based on the predictor to control the occurrence frequency of the behaviors, making the synthesized speech vary from less disfluent to more disfluent.To model the speech entrainment at acoustic level, we utilize a context acoustic encoder to extract a global style embedding from the previous speech conditioning on the synthesizing of current speech.Furthermore, since the current and previous utterances belong to the different speakers in a conversation, we add a domain adversarial training module to eliminate the speaker-related information in the acoustic encoder while maintaining the style-related information.Experiments show that our proposed approach can synthesize realistic conversations and control the occurrences of the spontaneous behaviors naturally.

Recurrent neural networks, such as gated recurrent units (GRUs) and long short-term memory (LSTM), are widely used on acoustic modeling for speech synthesis. However, such sequential generating processes are not friendly to today’s massively parallel computing devices. We introduce a fully convolutional neural network (CNN) model, which can effiently run on parallel processers, for speech synthesis. To improve the quality of the generated acoustic features, we strengthen our model with variational inference. We also use quasi-recurrent neural networks (QRNNs) to smoothen the generated acoustic features. Finally, a high-quality parallel WaveNet model is used to generate audio samples. Our contributions are twofold. First, we show that CNNs with variational inference can generate highly natural speech on a par with end-to-end models; the use of QRNNs further improves the synthetic quality by reducing trembling of generated acoustic features and introduces very little run-time overheads. Second, we show some techniques to further speed up the sampling process of the parallel WaveNet model.

Bulk transfer on the wide-area network (WAN) is a fundamental service to many globally-distributed applications. It is challenging to efficiently utilize expensive WAN bandwidth to achieve short transfer completion time and meet mission-critical deadlines. Advancements in software-defined networking (SDN) and optical hardware make it feasible and beneficial to quickly reconfigure optical devices in the optical layer, which brings a new opportunity for traffic management on the WAN.

With recent standardization and deployment of LTE eMBMS, cellular multicast is gaining traction as a method of efficiently using wireless spectrum to deliver large amounts of multimedia data to multiple cell sites. Cellular operators still seek methods of performing optimal resource allocation in eMBMS based on a complete understanding of the complex interactions among a number of mechanisms: the multicast coding scheme, the resources allocated to unicast users and their scheduling at the base stations, the resources allocated to a multicast group to satisfy the user experience of its members, and the number of groups and their membership, all of which we consider in this work. We determine the optimal allocation of wireless resources for users to maximize proportional fair utility. To handle the heterogeneity of user channel conditions, we efficiently and optimally partition multicast users into groups so that users with good signal strength do not suffer by being grouped together with users of poor signal strength. Numerical simulations are performed to compare our scheme to practical heuristics and state-of-the-art schemes. We demonstrate the tradeoff between improving unicast user rates and improving spectrum efficiency through multicast. Finally, we analyze the interaction between the globally fair solution and individual user's desire to maximize its rate. We show that even if the user deviates from the global solution in a number of scenarios, we can bound the number of selfish users that will choose to deviate.

The rapid advancement of smartphones has instigated tremendous data applications for cell phones. Supporting simultaneous voice and data services in a cellular network is not only desirable but also becoming indispensable. However, if the voice and data are serviced through the same antenna (like the 3G UMTS network), a voice call with data sessions requires better radio connection than a voice-only call. In this paper, we systematically study the coordination between the voice and data transmissions in UMTS networks. From analyzing a large carrier's UMTS network recording data, we first identify the most relevant network measurements/features indicating a potential call drop, then propose a drop-call predictor based on AdaBoost. Moreover, we develop an intelligent call management strategy to voluntarily block data sessions when the voice is predicted to be dropped. Our analysis utilizing real service provider's data sets shows that our proposed scheme can not only predict drop calls with a very high accuracy but also achieve the highest user satisfaction compared to the other existing call management strategies.

Metropolitan (metro) carriers are deploying next-generation metro Ethernet networks. This deployment will eventually replace traditional private-line services provided by legacy Time Division Multiplexing (TDM) technologies (such as Digital Cross-Connect Systems and SONET rings) with metro Ethernet services. A critical near-term need is the enabling of rich wireless data applications and, thus, begins a massive upgrade of backhaul links from cellular base stations to their Mobile Telephone Switching Offices (MTSOs), where the various voice and data applications will be provided. These base stations are evolving to 3G/4G technology and require metro Ethernet transport and at significantly higher backhaul technology than the prevalent TDM DS1s (1.5 Mb/s). In this paper, we describe a methodology to enable the rapid introduction of metro Ethernet networks, which includes architecture planning, switch location selection, link placement and sizing, as well as customization for cellular network reliability. Our approach uses a combination of many optimization algorithms and has been integrated into a pragmatic tool used by AT&T network planners. Case studies in very large US metro areas have shown that the tool gives cost-ffective solutions consistent with planner expectations and intuition.

Recent deployment of IP based multimedia distribution, especially broadcast TV distribution has increased the importance of simple and fast restoration during IP network failures for service providers. In this paper, we propose and evaluate a simple but efficient method for fast rerouting of IP multicast traffic during link failures in managed IPTV networks. More specifically, we devise an algorithm for tuning IP link weights so that the multicast routing path and the unicast routing path between any two routers are failure disjoint, allowing us to use unicast IP encapsulation for undelivered multicast packets during link failures. We demonstrate that, our method can be realized with minor modification to the current multicast routing protocol (PIM-SM). We run our prototype implementation in Emulab which shows our method yields to good performance.

In a network with Reconfigurable Optical Add/drop Multiplexers (ROADMs), transient effects from network failures trigger sudden power changes to channels, even on spans (ROADM-to-ROADM links) upstream/downstream from the failure. Some ROADMs rely on restrictive wavelength assignment rules, calledfield-of-use(FOU), to insure surviving channels meet performance objectives.Dropdepth(DD) (ratio of surviving channels) andspanoverlapcount(SOC) (length of spans of overlapping lightpaths) are two parameters defined for FOU guidelines. For a given DD, FOU guidelines specify a corresponding maximum SOC for each wavelength. Generally, a wavelength can support a longer span overlap count under a smaller drop depth restriction. However, the measured drop-depth changes as wavelengths are assigned. We propose methods to optimize transient-constrained wavelength assignment in commercial ROADM networks. Our solutions have been shown to provide 40% more channels than with typical FOU guidelines.

Multimedia distribution, especially broadcast TV distribution over an IP network requires high bandwidth combined with tight latency and loss constraints, even under failure conditions. Due to the high bandwidth requirements of broadcast TV distribution, use of IP-based multicast to distribute TV content is needed for efficient use of capacity. The protection and restoration mechanisms currently adopted in IP backbones use either IGP re-convergence or some form of fast reroute. The IGP re-convergence mechanism is too slow for real-time multimedia distribution while a drawback of fast reroute is that traffic is re-routed on a link-basis (instead of end-to-end) so there can be traffic overlap during failures. By this we mean traffic passing through the same link along the same direction more than once; this requires more link capacity or it will result in congestion. We propose a routing method that interacts with Fast Reroute and multicast to minimize traffic overlap during failures. We also present an algorithm for link weight setting that avoids traffic overlap due to any single link failure. Performance analysis shows that our methods improve network service availability and significantly reduce the impact of failures.

With more service providers making considerable investments to roll out multimedia services using IP technology, live TV distribution on IP network is expected to grow impressively over next few years. To make efficient use of IP network infrastructure, service providers use multicast to transmit broadcast video content to the receiving nodes while simultaneously using unicast to transport other services on the same network, such as video on demand (VoD), high-speed Internet (HIS) etc. How to engineer different traffic flows on the same network to avoid traffic congestion becomes a major design issue. Although the service traffic flows are from source to receivers, the IP multicast tree is calculated using IGP shortest path from the receivers to the source (backwards from the flow), while the unicast path is calculated from the source to the receivers (same direction as the flow). To minimize congestion, we take advantage of this property and propose an algorithm to tune IGP link weights (such as OSPF) such that the traffic flows for multicast and unicast don't overlap. This proposal provides a natural way for service providers to use the spare capacity on the opposite direction of multicast traffic flow to roll out additional services in the same IP networks without impacting existing multicast services.

We describe our implementation choices and experiences gained from developing a GMPLS-based control plane test-bed prototype for end-lo-end services. The prototype includes control plane components of routing and signaling protocols and provides rapid provisioning and restoration of connections across multiple optical network domains. Our experience in implementing the multi-domain signaling showed that it was possible to simply adapt the signaling schemes within the OIF UNI specifications. Our test-bed measurements achieved average 27ms provisioning time and 16.3 ms restoration time for connections crossing three domains.

IP over reconfigurable optical network architectures have been extensively discussed within the research literature over the past few years. However, although reconfigurable optical networks have been deployed and signaling protocols between IP routers and optical networks have been standardized, large IP backbones are typically deployed using the reconfigurable optical networks. One of the most important criteria in determining whether an IP backbone should be carried over a reconfigurable optical network is economic viability - which necessitates a detailed, accurate economic study of IP backbone over reconfigurable optical network architectures. In this paper, we analyze and explore four IP over optical network architectures for a typical large ISP backbone. In contrast with other published claims, our results suggest that an IP over opaque reconfigurable optical network architecture is not economically attractive with current equipment and IP backbone network design requirements. However, for ISPs also carrying large volumes of transport network private line services, our proposed integrated IP over re-configurable optical network architecture may provide an attractive alternative for providing rapid, cost effective failure recovery.

Multiple addresses within an OSPF area can be aggregated and advertised together to other areas. This process is known as address aggregation and is used to reduce router computational overheads and memory requirements and to reduce the network bandwidth consumed by OSPF messages. The downside of address aggregation is that it leads to information loss and consequently sub-optimal (non-shortest path) routing of data packets. The resulting difference (path selection error) between the length of the actual forwarding path and the shortest path varies between different sources and destinations. This paper proves that the path selection error from any source to any destination can be bounded using only parameters describing the destination area. Based on this, the paper presents an efficient algorithm that generates the minimum number of aggregates subject to a maximum allowed path selection error. A major operational benefit of our algorithm is that network administrators can select aggregates for an area based solely on the topology of the area without worrying about remaining areas of the OSPF network. The other benefit is that the algorithm enables trade-offs between the number of aggregates and the bound on the path selection error. The paper also evaluates the algorithm's performance on random topologies. Our results show that in some cases, the algorithm is capable of reducing the number of aggregates by as much as 50% with only a relatively small introduction of maximum path selection error.

In MPLS/GMPLS networks, a range of restoration schemes will be required to support different tradeoffs between service interruption time and network resource utilization. In light of these tradeoffs, path-based end-to-end shared mesh restoration provides a very attractive solution. However, efficient use of bandwidth for shared mesh restoration strongly relies on the procedure for selecting restoration paths. We propose an efficient restoration path selection algorithm for restorable connections over shared bandwidth in a fully distributed MPLS/GMPLS architecture. We also describe how to extend MPLS/GMPLS signaling protocols to collect the necessary information efficiently. To evaluate the algorithm's performance, we compare it via simulation with two other well-known algorithms on a typical intercity backbone network. The key figure of merit for restoration bandwidth efficiency is restoration overbuild, i.e., the extra bandwidth required to meet the network restoration objective as a percentage of the bandwidth of the network with no restoration. Our simulation results show that our algorithm uses significantly less restoration overbuild (63%-68%) compared with the other two algorithms (83%-90%).

In MPLS/GMPLS networks, a range of restoration schemes are required to support different tradeoffs between service interruption time and network resource utilization. In light of these tradeoffs, path-based, end-to-end shared restoration provides a very attractive solution. However, efficient use of capacity for shared restoration strongly relies on the selection procedure of restoration paths. We propose an efficient path-selection algorithm for restoration of connections over shared bandwidth in a fully distributed GMPLS architecture. We also describe how to extend GMPLS signaling protocols to collect the necessary information efficiently. To evaluate the algorithm's performance, we compare it via simulation with two other well-known algorithm on a typical intercity backbone network. The key figure-of-merit for restoration capacity efficiency is restoration overbuild, i.e., the extra capacity required to meet the network restoration objective as a percentage of the capacity of the network with no restoration. Our simulation results show that our algorithm uses significantly less restoration overbuild (63-68%) compared to the other two algorithms (83-90%).

This paper studies-the off-line wavelength assignment problem in star and ring networks that deploy multiple fibers between nodes and use wavelength division multiplexing (WDM) for transmission. The results in this paper show that the ability to switch between fibers increases wavelength utilization. In particular, sharper per-fiber bounds on the number of required wavelengths are derived for the multifiber version of the assignment problem in star and ring networks. Additionally, the complexity of the problem is studied and several constrained versions of the problem are also considered for star and ring networks. A summary of contributions is provided.

This paper studies the off-line wavelength assignment problem in star and ring networks that deploy multiple fibers between nodes and use wavelength division multiplexing (WDM) for transmission. The results in this paper show that the ability to switch between fibers increases wavelength utilization. In particular, sharper per-fiber bounds on the number of required wavelengths are derived for the multifiber version of the assignment problem in star and ring networks. Additionally, the complexity of the problem is studied and several constrained versions of the problem are also considered for star and ring networks. A summary of contributions is provided in the first section.

Solving systems of fuzzy relation equations is an important topic in fuzzy set theory. This paper studies the composite interval-valued fuzzy relation equations. After analyzing the properties of its solution set, we convert the fuzzy relation equations into a fuzzy relation inequality system and propose an efficient computational procedure to generate the whole solution set. Examples are included to illustrate the idea and algorithm.