In this paper, we propose a joint federated learning (FL) and personalization method for on-device ASR adaptation. Starting with a Conformer-based RNN-T as the ASR model backbone that is pretrained on public data and shared across devices, we propose to adapt to user data on-device by ① collectively finetuning the backbone on all user data by FL, and ② for each user, augmenting the backbone with a personalized adapter that is trained on on-device data and stored locally on their devices. As ground-truth transcriptions are not available, we use pseudo-label training, which can be completely performed on-device. Our joint method combines the best of both FL and personalization and achieves maximum effect for both heavy and light users: For users with 50+ adaptation utterances, our recipe achieves $27.4 \%$ relative WER reduction, largely due to personalization; for users with no adaptation utterance, our recipe achieves $8.9 \%$ relative WER reduction purely due to FL.

Scalability and privacy are two critical concerns for cross-device federated learning (FL) systems. In this work, we identify that synchronous FL – cannot scale efficiently beyond a few hundred clients training in parallel. It leads to diminishing returns in model performance and training speed, analogous to large-batch training. On the other hand, asynchronous aggregation of client updates in FL (i.e., asynchronous FL) alleviates the scalability issue. However, aggregating individual client updates is incompatible with Secure Aggregation, which could result in an undesirable level of privacy for the system. To address these concerns, we propose a novel buffered asynchronous aggregation method, FedBuff, that is agnostic to the choice of optimizer, and combines the best properties of synchronous and asynchronous FL. We empirically demonstrate that FedBuff is $3.3\times$ more efficient than synchronous FL and up to $2.5\times$ more efficient than asynchronous FL, while being compatible with privacy-preserving technologies such as Secure Aggregation and differential privacy. We provide theoretical convergence guarantees in a smooth non-convex setting. Finally, we show that under differentially private training, FedBuff can outperform FedAvgM at low privacy settings and achieve the same utility for higher privacy settings.

We consider the privacy-preserving machine learning (ML) setting where the trained model must satisfy differential privacy (DP) with respect to the labels of the training examples. We propose two novel approaches based on, respectively, the Laplace mechanism and the PATE framework, and demonstrate their effectiveness on standard benchmarks.While recent work by Ghazi et al. proposed Label DP schemes based on a randomized response mechanism, we argue that additive Laplace noise coupled with Bayesian inference (ALIBI) is a better fit for typical ML tasks. Moreover, we show how to achieve very strong privacy levels in some regimes, with our adaptation of the PATE framework that builds on recent advances in semi-supervised learning.We complement theoretical analysis of our algorithms' privacy guarantees with empirical evaluation of their memorization properties. Our evaluation suggests that comparing different algorithms according to their provable DP guarantees can be misleading and favor a less private algorithm with a tighter analysis.Code for implementation of algorithms and memorization attacks is available from https://github.com/facebookresearch/labeldpantipodes.

A local fingerprinting algorithm is proposed for the purpose of audio copy detection. The proposed algorithm is robust to noise as well as tempo and pitch modifications of the audio signal. The fingerprints are extracted from adaptively scaled patches of the time-chroma representation of the audio signal. The proposed time-chroma representation, converts tempo change and pitch shift attacks on an audio signal to scaling and circular shift attacks on images, respectively. The proposed algorithm is shown to outperform the state-of-the-art.

A fast approximate nearest neighbor search algorithm for the (binary) Hamming space is proposed. The proposed Error Weighted Hashing (EWH) algorithm is up to 20 times faster than the popular locality sensitive hashing (LSH) algorithm and works well even for large nearest neighbor distances where LSH fails. EWH significantly reduces the number of candidate nearest neighbors by weighing them based on the difference between their hash vectors. EWH can be used for multimedia retrieval and copy detection systems that are based on binary fingerprinting. On a fingerprint database with more than 1,000 videos, for a specific detection accuracy, we demonstrate that EWH is more than 10 times faster than LSH. For the same retrieval time, we show that EWH has a significantly better detection accuracy with a 15 times lower error rate.

A video copy detection system that is based on content fingerprinting and can be used for video indexing and copyright applications is proposed. The system relies on a fingerprint extraction algorithm followed by a fast approximate search algorithm. The fingerprint extraction algorithm extracts compact content-based signatures from special images constructed from the video. Each such image represents a short segment of the video and contains temporal as well as spatial information about the video segment. These images are denoted by temporally informative representative images. To find whether a query video (or a part of it) is copied from a video in a video database, the fingerprints of all the videos in the database are extracted and stored in advance. The search algorithm searches the stored fingerprints to find close enough matches for the fingerprints of the query video. The proposed fast approximate search algorithm facilitates the online application of the system to a large video database of tens of millions of fingerprints, so that a match (if it exists) is found in a few seconds. The proposed system is tested on a database of 200 videos in the presence of different types of distortions such as noise, changes in brightness/contrast, frame loss, shift, rotation, and time shift. It yields a high average true positive rate of 97.6% and a low average false positive rate of 1.0%. These results emphasize the robustness and discrimination properties of the proposed copy detection system. As security of a fingerprinting system is important for certain applications such as copyright protections, a secure version of the system is also presented.

Content-based video hashing was introduced recently to serve the purpose of video copy detection. A conventional approach to video hashing is to apply image hashing techniques to either every frame or to the selected key frames of a video sequence. Both approaches ignore the temporal information contained in a video sequence. This study proposes an approach for generating representative images of a video sequence that carry the temporal as well as the spatial information. These images are denoted as TIRIs, Temporally Informative Representative Images. Performance of the proposed approach is demonstrated by applying a simple image hashing technique on TIRIs of a video database. It is shown that the resulted video hashing algorithm is highly robust to noise, frame dropping, changes in brightness and contrast, as well as a range of geometric attacks. An average true positive rate of 99.2% and false positive rate of 0.4% of the proposed approach demonstrate the robustness and uniqueness of the generated hashes. It is demonstrated that the proposed approach is easy to implement and computationally more efficient than another state-of-the-art video hashing method.