While crowdsourcing is an established solution for facilitating and scaling the collection of speech data, the involvement of non-experts necessitates protocols to ensure final data quality. To reduce the costs of these essential controls, this paper investigates the use of Speech Foundation Models (SFMs) to automate the validation process, examining for the first time the cost/quality trade-off in data acquisition. Experiments conducted on French, German, and Korean data demonstrate that SFM-based validation has the potential to reduce reliance on human validation, resulting in an estimated cost saving of over 40.0% without degrading final data quality. These findings open new opportunities for more efficient, cost-effective, and scalable speech data acquisition.

Fine-tuning can be used to adapt an Automatic Speech Recognition system to a new domain, based on some transcribed data from the target-domain. However, in real-world settings, the availability and the amount of target-domain data needed for this fine-tuning can be limited, because of budget constraints or other reasons like privacy. In such cases, a possible approach is to automatically select candidate training data from a pre-existing pool of audio data (e.g. a mix of open-source datasets), based on a sample of the target domain. In this paper we investigate unsupervised data selection techniques for fine-tuning, under a limited budget of only one hour of training data, using a multi-source and multi-domain pool of data (7 datasets, 6k hours, various genres and styles).Our method consists in 1. extracting self-supervised model representations of both modalities (text and audio) 2. learning from these representations a domain-calibrated vector representation of what a domain is in terms of origin, genre and style and finally 3. using this representation with k-nearest neighbor search for automatic data selection. Using this method we observe an improvement of the Word Error Rate of up to about 13% (3.2 WER points) on average, compared to our baselines.

Self-supervised learning models have revolutionized the field of speech processing. However, the process of fine-tuning these models on downstream tasks requires substantial computational resources, particularly when dealing with multiple speech-processing tasks. In this paper, we explore the potential of adapter-based fine-tuning in developing a unified model capable of effectively handling multiple spoken language processing tasks. The tasks we investigate are Automatic Speech Recognition, Phoneme Recognition, Intent Classification, Slot Filling, and Spoken Emotion Recognition. We validate our approach through a series of experiments on the SUPERB benchmark, and our results indicate that adapter-based fine-tuning enables a single encoder-decoder model to perform multiple speech processing tasks with an average improvement of 18.4 % across the five target tasks while staying efficient in terms of parameter updates.

In this paper, we look at the case of a Generic text-to-text NMT model that has to deal with data coming from various modalities, like speech, images, or noisy text extracted from the web.We propose a two-step method, based on composable adapters, to deal with this problem of Multimodal Robustness.In the first step, we separately learn domain adapters and modality specific adapters, to deal with noisy input coming from various sources: ASR, OCR, or noisy text (UGC).In a second step, we combine these components at runtime via dynamic routing or, when the source of noise is unknown, via two new transfer learning mechanisms (Fast Fusion and Multi Fusion).We show that our method provides a flexible, state-of-the-art, architecture able to deal with noisy multimodal inputs.

We are also grateful to our

Applications that process Point-of-Interest data are omnipresent nowadays. They range from digital maps to recommender systems for places to visit, and personal assistants. The success of such applications critically depends on the quality of the ingested data. However, corresponding databases, especially when they are crowdsourced, are often incomplete. Existing work on automatic data completion approaches has only partially considered the task for Points-of-Interest (POI). Such entities have a number of distinctive properties – notably multiscript names, geo-spatial identity, and temporally defined context –, which make the task more challenging. Here we present an approach to automatically complete POI semantic tags in a crowdsourced database. We perform experiments on multi-lingual data from Foursquare, a global location-based social network, and observe that (i) POI names are strong predictors of POI semantic tags: a character-based LSTM model trained only on POI names gives 72.5% worst-case micro-precision and 50.38% micro F1 scores, (ii) appropriate use of spatio-temporal data leads to consistent improvements (iii) using a structured representation of time gives higher precision and requires less computation time than string-based LSTM variants, however, the higher precision is achieved at a cost of lower recall and micro-F1, (iv) an LSTM model trained on semi-structured strings representing time, is competitive to fully structured inputs in terms of recall.