Existing large-scale zero-shot text-to-speech (TTS) models deliver high speech quality but suffer from slow inference speeds due to massive parameters. To address this issue, this paper introduces ZipVoice, a high-quality flow-matching-based zero-shot TTS model with a compact model size and fast inference speed. Key designs include: 1) a Zipformer-based vector field estimator to maintain adequate modeling capabilities under constrained size; 2) Average upsampling-based initial speech-text alignment and Zipformer-based text encoder to improve speech intelligibility; 3) A flow distillation method to reduce sampling steps and eliminate the inference overhead associated with classifier-free guidance. Experiments on 100 k hours multilingual datasets show that ZipVoice matches state-of-the-art models in speech quality, while being 3 times smaller and up to 30 times faster than a DiT-based flow-matching baseline. Codes, model checkpoints and demo samples are publicly available.11https://github.com/k2-fsa/ZipVoice

Connectionist Temporal Classification (CTC) is a widely used method for automatic speech recognition (ASR), renowned for its simplicity and computational efficiency. However, it often falls short in recognition performance. In this work, we propose the Consistency-Regularized CTC (CR-CTC), which enforces consistency between two CTC distributions obtained from different augmented views of the input speech mel-spectrogram. We provide in-depth insights into its essential behaviors from three perspectives: 1) it conducts self-distillation between random pairs of sub-models that process different augmented views; 2) it learns contextual representation through masked prediction for positions within time-masked regions, especially when we increase the amount of time masking; 3) it suppresses the extremely peaky CTC distributions, thereby reducing overfitting and improving the generalization ability. Extensive experiments on LibriSpeech, Aishell-1, and GigaSpeech datasets demonstrate the effectiveness of our CR-CTC. It significantly improves the CTC performance, achieving state-of-the-art results comparable to those attained by transducer or systems combining CTC and attention-based encoder-decoder (CTC/AED). We release our code at \url{https://github.com/k2-fsa/icefall}.

Self-supervised learning (SSL) has achieved great success in speech-related tasks. While Transformer and Conformer architectures have dominated SSL backbones, encoders like Zipformer, which excel in automatic speech recognition (ASR), remain unexplored in SSL. Concurrently, inefficiencies in data processing within existing SSL training frameworks, such as fairseq, pose challenges in managing the growing volumes of training data. To address these issues, we propose k2SSL, an open-source framework that offers faster, more memory-efficient, and better-performing self-supervised speech representation learning, focusing on downstream ASR tasks. The optimized HuBERT and proposed Zipformer-based SSL systems exhibit substantial reductions in both training time and memory usage during SSL training. Experiments on LibriSpeech demonstrate that Zipformer Base significantly outperforms HuBERT and WavLM, achieving up to a 34.8% relative WER reduction compared to HuBERT Base after fine-tuning, along with a 3.5x pre-training speedup in GPU hours. When scaled to 60k hours of LibriLight data, Zipformer Large exhibits remarkable efficiency, matching HuBERT Large’s performance while requiring only 5/8 pre-training steps.

In this paper, we introduce Libriheavy, a large-scale ASR corpus consisting of 50,000 hours of read English speech derived from LibriVox. To the best of our knowledge, Libriheavy is the largest freely-available corpus of speech with supervisions. Different from other open-sourced datasets that only provide normalized transcriptions, Libriheavy contains richer information such as punctuation, casing and text context, which brings more flexibility for system building. Specifically, we propose a general and efficient pipeline to locate, align and segment the audios in previously published Librilight to its corresponding texts. The same as Librilight, Libriheavy also has three training subsets small, medium, large of the sizes 500h, 5000h, 50000h respectively. We also extract the dev and test evaluation sets from the aligned audios and guarantee there is no overlapping speakers and books in training sets. Baseline systems are built on the popular CTC-Attention and transducer models. Additionally, we open-source our dataset creatation pipeline which can also be used to other audio alignment tasks.

Prompts are crucial to large language models as they provide context information such as topic or logical relationships. Inspired by this, we propose PromptASR, a framework that integrates prompts in end-to-end automatic speech recognition (E2E ASR) systems to achieve contextualized ASR with controllable style of transcriptions. Specifically, a dedicated text encoder encodes the text prompts and the encodings are injected into the speech encoder by cross-attending the features from two modalities. When using the ground truth text from preceding utterances as content prompt, the proposed system achieves 21.9% and 6.8% relative word error rate reductions on a book reading dataset and an in-house dataset compared to a baseline ASR system. The system can also take word-level biasing lists as prompt to improve recognition accuracy on rare words. An additional style prompt can be given to the text encoder and guide the ASR system to output different styles of transcriptions. The code is available at icefall1.

Cross-modality knowledge transfer aims to apply knowledge learned in the source modality to the target modality. It is more challenging than the general knowledge transfer task because of the aggravated modality shift problem due to introducing heterogeneous data. This paper proposes a novel fast cross-modality knowledge transfer method via a contextual autoencoder transformation. In particular, the encoder projects the contextual representations of the source modality into the target modality. Then to bridge the semantic shared among source and target modalities, the decoder exerts an additional constraint to reconstruct the original source modality. We show that this constraint is beneficial for mitigating the shift problem and improves the generalization from heterogeneous modalities. Remarkably, the autoencoder is linear and symmetric, facilitating scalability for large-scale datasets. Experimental results on two widely used benchmarks demonstrate that the proposed method surpasses several state-of-the-arts baselines, validating its effectiveness and efficiency.

The Conformer has become the most popular encoder model for automatic speech recognition (ASR). It adds convolution modules to a transformer to learn both local and global dependencies. In this work we describe a faster, more memory-efficient, and better-performing transformer, called Zipformer. Modeling changes include: 1) a U-Net-like encoder structure where middle stacks operate at lower frame rates; 2) reorganized block structure with more modules, within which we re-use attention weights for efficiency; 3) a modified form of LayerNorm called BiasNorm allows us to retain some length information; 4) new activation functions SwooshR and SwooshL work better than Swish. We also propose a new optimizer, called ScaledAdam, which scales the update by each tensor's current scale to keep the relative change about the same, and also explictly learns the parameter scale. It achieves faster converge and better performance than Adam. Extensive experiments on LibriSpeech, Aishell-1, and WenetSpeech datasets demonstrate the effectiveness of our proposed Zipformer over other state-of-the-art ASR models. Our code is publicly available at https://github.com/k2-fsa/icefall.

Knowledge distillation (KD) is a common approach to improve model performance in automatic speech recognition (ASR), where a student model is trained to imitate the output behaviour of a teacher model. However, traditional KD methods suffer from teacher label storage issue, especially when the training corpora are large. Although on-the-fly teacher label generation tackles this issue, the training speed is significantly slower as the teacher model has to be evaluated every batch. In this paper, we reformulate the generation of teacher label as a codec problem. We propose a novel Multi-codebook Vector Quantization (MVQ) approach that compresses teacher embeddings to codebook indexes (CI). Based on this, a KD training framework (MVQ-KD) is proposed where a student model predicts the CI generated from the embeddings of a self-supervised pre-trained teacher model. Experiments on the LibriSpeech clean-100 hour show that MVQ-KD framework achieves comparable performance as traditional KD methods (11, 12), while requiring 256 times less storage. When the full LibriSpeech dataset is used, MVQ-KD framework results in 13.8% and 8.2% relative word error rate reductions (WERRs) for non -streaming transducer on test-clean and test-other and 4.0% and 4.9% for streaming transducer. The implementation of this work is already released as a part of the open-source project icefall1.

The transducer architecture is becoming increasingly popular in the field of speech recognition, because it is naturally streaming as well as high in accuracy. One of the drawbacks of transducer is that it is difficult to decode in a fast and parallel way due to an unconstrained number of symbols that can be emitted per time step.In this work, we introduce a constrained version of transducer loss to learn strictly monotonic alignments between the sequences; we also improve the standard greedy search and beam search algorithms by limiting the number of symbols that can be emitted per time step in transducer decoding, making it more efficient to decode in parallel with batches. Furthermore, we propose an finite state automaton-based (FSA) parallel beam search algorithm that can run with graphs on GPU efficiently. The experiment results show that we achieve slight word error rate (WER) improvement as well as significant speedup in decoding. Our work is open-sourced and publicly available1.

In streaming automatic speech recognition (ASR), it is desirable to reduce latency as much as possible while having minimum impact on recognition accuracy. Although a few existing methods are able to achieve this goal, they are difficult to implement due to their dependency on external alignments. In this paper, we propose a simple way to penalize symbol delay in transducer model, so that we can balance the trade-off between symbol delay and accuracy for streaming models without external alignments. Specifically, our method adds a small constant times (T/2 - t), where T is the number of frames and t is the current frame, to all the non-blank log-probabilities (after normalization) that are fed into the two dimensional transducer recursion. For both streaming Conformer models and unidirectional long short-term memory (LSTM) models, experimental results show that it can significantly reduce the symbol delay with an acceptable performance degradation. Our method achieves similar delay-accuracy trade-off to the previously published FastEmit, but we believe our method is preferable because it has a better justification: it is equivalent to penalizing the average symbol delay. Our work is open-sourced and publicly available1.

The RNN-Transducer (RNN-T) framework for speech recognition has been growing in popularity, particularly for deployed real-time ASR systems, because it combines high accuracy with naturally streaming recognition.One of the drawbacks of RNN-T is that its loss function is relatively slow to compute, and can use a lot of memory.Excessive GPU memory usage can make it impractical to use RNN-T loss in cases where the vocabulary size is large: for example, for Chinese character-based ASR.We introduce a method for faster and more memoryefficient RNN-T loss computation.We first obtain pruning bounds for the RNN-T recursion using a simple joiner network that is linear in the encoder and decoder embeddings; we can evaluate this without using much memory.We then use those pruning bounds to evaluate the full, non-linear joiner network.The code is open-sourced and publicly available.

Speech emotion recognition (SER) is becoming the main human–computer interaction logic for autonomous vehicles in the next generation of intelligent transportation systems (ITSs). It can improve not only the safety of autonomous vehicles but also the personalized in-vehicle experience. However, current vehicle-mounted SER systems still suffer from two major shortcomings. One is the insufficient service capacity of the vehicle communication network, which is unable to meet the SER needs of autonomous vehicles in next-generation ITSs in terms of the data transmission rate, power consumption, and latency. Second, the accuracy of SER is poor, and it cannot provide sufficient interactivity and personalization between users and vehicles. To address these issues, we propose an SER-enhanced traffic efficiency solution for autonomous vehicles in a 5G-enabled space–air–ground integrated network (SAGIN)-based ITS. First, we convert the vehicle speech information data into spectrograms and input them into an AlexNet network model to obtain the high-level features of the vehicle speech acoustic model. At the same time, we convert the vehicle speech information data into text information and input it into the Bidirectional Encoder Representations from Transformers (BERT) model to obtain the high-level features of the corresponding text model. Finally, these two sets of high-level features are cascaded together to obtain fused features, which are sent to a softmax classifier for emotion matching and classification. Experiments show that the proposed solution can improve not only the SAGIN’s service capabilities, resulting in a large capacity, high bandwidth, ultralow latency, and high reliability, but also the accuracy of vehicle SER as well as the performance, practicality, and user experience of the ITS

It is expected that a mixture of autonomous and manual vehicles will persist as a part of the intelligent transportation system (ITS) for many decades. Thus, addressing the safety issues arising from this mix of autonomous and manual vehicles before autonomous vehicles are entirely popularized is crucial. As the ITS system has increased in complexity, autonomous vehicles exhibit problems such as a low intention recognition rate and poor real-time performance when predicting the driving direction; these problems seriously affect the safety and comfort of mixed traffic systems. Therefore, the ability of autonomous vehicles to predict the driving direction in real time according to the surrounding traffic environment must be improved and researchers must work to create a more mature ITS. In this paper, we propose a deep learning-based traffic safety solution for a mixture of autonomous and manual vehicles in a 5G-enabled ITS. In this scheme, a driving trajectory dataset and a natural-driving dataset are employed as the network inputs to long-term memory networks in the 5G-enabled ITS: the probability matrix of each intention is calculated by the softmax function. Then, the final intention probability is obtained by fusing the mean rule in the decision layer. Experimental results show that the proposed scheme achieves intention recognition rates of 91.58% and 90.88% for left and right lane changes, respectively, effectively improving both accuracy and real-time intention recognition and improving the lane change problem in a mixed traffic environment.

Choosing the best cloud configuration for large-scale data analytics jobs deployed in the cloud can substantially improve their performance and reduce costs. However, current cloud providers offer a wide variety of instance types and customized cluster sizes, making it both time-consuming and costly to pinpoint the optimal cloud configuration. This article presents the design, implementation, and evaluation of Silhouette, a cloud configuration selection framework based on performance models for various large-scale analytics jobs with minimal training overhead. The essence of Silhouette is to build performance prediction models with carefully selected small-scale experiments on small subsets of input data to estimate the performance with entire input data on larger cluster sizes. To reduce the training time and cost, Silhouette incorporates new statistical techniques to select those experiments that yield the best possible information for performance prediction. Moreover, we develop a novel model transformer to convert a prediction model built on one instance type to a different instance type with only one extra experiment, which significantly reduces the training overhead. We evaluate Silhouette with an extensive array of large-scale data analytics jobs on Amazon EC2. Our experimental results have shown convincing evidence that Silhouette is effective in optimizing cloud configuration while saving both training time and costs compared with existing solutions.

Recently, there have been several high-profile achievements of agents learning to play games against humans and beat them. In this article, we study the problem of training intelligent agents in service of game development. Unlike the agents built to “beat the game,” our agents aim to produce human-like behavior to help with game evaluation and balancing. We discuss two fundamental metrics based on which we measure the human-likeness of agents, namely skill and style, which are multifaceted concepts with practical implications outlined in this article. We report four case studies in which the style and skill requirements inform the choice of algorithms and metrics used to train agents; ranging from A* search to state-of-the-art deep reinforcement learning (RL). Furthermore, we, show that the learning potential of state-of-the-art deep RL models does not seamlessly transfer from the benchmark environments to target ones without heavily tuning their hyperparameters, leading to linear scaling of the engineering efforts, and computational cost with the number of target domains.

The dramatic growth of mobile application usage has posed great pressure on application developers to better manage their backend capacity. Rule-based or schedule-based auto-scaling mechanisms have been proposed, but it is difficult or expensive to frequently adjust the backend capacity to track the burstiness of mobile traffic. In this paper, we explore a fundamentally different approach. Instead of scaling the backend in line with the mobile traffic, we smooth out traffic profiles to reduce the required backend capacity and increase its utilization. Our proposed solution, called Razor, is inspired by two key insights on mobile traffic. First, mobile traffic exhibits high short-term fluctuations but steady long-term trend, so that we may temporarily delay user requests and periodically adapt backend capacity based on the predicted traffic volume. Second, user requests have different priorities: while some requests are urgent (e.g., sending a message), some are delay-tolerant (e.g., changing the profile photo) and can be postponed without much influence on the user experience. Based on these observations, our design features a two-tier architecture: on a long timescale, Razor predicts future traffic using machine learning algorithms and plans the optimal backend capacity to minimize the budget with performance guarantee; on a short timescale, Razor schedules which requests to delay and by how much time to delay according to their delay tolerance. We implement a fully-functional prototype of Razor, and evaluate its performance with both real and synthetic traces. Extensive experimental results show that Razor can effectively help mobile application developers reduce their backend cost while guaranteeing the user experience.

The use of a combinatorial auction is believed to be an effective way to distribute spectrum to buyers who have diversified valuations for different spectrum combinations. However, the allocation of spectrum with combinatorial auctions mainly aims at optimizing over certain utility functions, e.g., social welfare, but ignores individual preferences of buyers and sellers, who have incentives to deviate from globally optimal allocation results to improve their own utility. In this paper, we explore the possibility of designing a new stable matching algorithm for combinatorial spectrum allocations. Starkly different from existing efforts on spectrum matching mechanism design, our proposed combinatorial spectrum matching framework not only allows buyers to express preferences towards spectrum combinations (rather than individual channels), but also computes the payment that should be transferred from buyers to sellers. Payment determination, while essential in spectrum exchange, has never been addressed in existing spectrum matching frameworks. We design a novel algorithm to achieve a stable combinatorial spectrum matching and to compute the corresponding payment profiles. We conducted an extensive array of experiments to compare the performance of stable combinatorial spectrum matching with spectrum auctions. It is shown that the combinatorial spectrum matching sacrifices little allocation efficiency in terms of social welfare and spectrum utilization, but achieves a much higher individual buyer utility, which will incentivize buyers to participate and comply with the allocation results.

We consider domain subdivision algorithms for computing isotopic approximations of nonsingular curves represented implicitly by an equation f(X,Y)=0. Two algorithms in this area are from Snyder (1992) and Plantinga & Vegter (2004). We introduce a new algorithm that combines the advantages of these two algorithms: like Snyder, we use the parametrizability criterion for subdivision, and like Plantinga & Vegter we exploit non-local isotopy. We further extend our algorithm in two important and practical directions: first, we allow subdivision cells to be rectangles with arbitrary but bounded aspect ratios. Second, we extend the input domains to be regions R0 with arbitrary geometry and which might not be simply connected. Our algorithm halts as long as the curve has no singularities in the region, and intersects the boundary of R0 transversally. Our algorithm is also easy to implement exactly. We report on very encouraging preliminary experimental results, showing that our algorithms can be much more efficient than both Plantinga & Vegter's and Snyder's algorithms.

Key management is very crucial in a secure multicast system. The key storage of the group controller and group members, the communication cost and computation cost caused by joining/leaving members, are the determining factors for the performance of the key management system. A scheme is high-performed, if it has the optimal rekeying cost and the lower storage requirements. In order to get the high efficiency and security, a novel scheme (TOFT) based on threshold-based one-way function tree is proposed in this paper. The quad-tree and the threshold-key-mechanism are used in the scheme, which improves the performance of the key management system. We present the design principle, the realization protocols including keys generation and distribution, dynamic membership management. The TOFT scheme is compared with other protocols from the following four aspects: computation cost, communication cost, storage requirements, and security. Finally, we conclude that our scheme is more efficient than others.