Recent advances in Automatic Speech Recognition (ASR) have been largely fueled by massive speech corpora.However, extending coverage to diverse languages with limited resources remains a formidable challenge.This paper introduces Speech Back-Translation, a scalable pipeline that improves multilingual ASR models by converting large-scale text corpora into synthetic speech via off-the-shelf text-tospeech (TTS) models.We demonstrate that just tens of hours of real transcribed speech can effectively train TTS models to generate synthetic speech at hundreds of times the original volume while maintaining high quality.To evaluate synthetic speech quality, we develop an intelligibility-based assessment framework and establish clear thresholds for when synthetic data benefits ASR training.Using Speech Back-Translation, we generate more than 500,000 hours of synthetic speech in ten languages and continue pre-training Whisperlarge-v3, achieving average transcription error reductions of over 30%.These results highlight the scalability and effectiveness of Speech Back-Translation for enhancing multilingual ASR systems.

Teaching small-scale language models to perform math reasoning is a valuable yet challenging task.Besides obtaining labeled data from human experts, one of the most common ways to collect high-quality data is by sampling from a larger and more powerful language model.Although previous works have demonstrated the effectiveness of this method, such a knowledge distillation paradigm can be costly and unstable, especially considering that many large language models, such as GPT-4 (Ope-nAI, 2023), are closed-source, proprietary, and their behaviors are unpredictable.In this work, to avoid relying on outputs from large models, we demonstrate that the reasoning abilities of small-scale language models can be enhanced through self-training, which involves training models with their own outputs.We also show that the conventional self-training can be further augmented by an alignment algorithm called Direct Preference Optimization (DPO) (Rafailov et al., 2023).We empirically found that models trained with the DPO objective are capable of making better generations that largely benefit multi-turn self-training.The experimental results show our models outperform the existing models with comparable sizes on the GSM8K benchmark with minimal resource requirements.1

Fine-tuning a pre-trained language model via the contrastive learning framework with a large amount of unlabeled sentences or labeled sentence pairs is a common way to obtain highquality sentence representations.Although the contrastive learning framework has shown its superiority on sentence representation learning over previous methods, the potential of such a framework is under-explored so far due to the simple method it used to construct positive pairs.Motivated by this, we propose a method that makes hard positives from the original training examples.A pivotal ingredient of our approach is the use of prefix that is attached to a pre-trained language model, which allows for differentiable data augmentation during contrastive learning.Our method can be summarized in two steps: supervised prefixtuning followed by joint contrastive fine-tuning with unlabeled or labeled examples.Our experiments confirm the effectiveness of our data augmentation approach.The proposed method yields significant improvements over existing methods under both semi-supervised and supervised settings.Our experiments under a low labeled data setting also show that our method is more label-efficient than the state-of-the-art contrastive learning methods. 1