Federated learning (FL) has been widely used to train neural networks with the decentralized training procedure where data is only accessed on clients’ devices for privacy preservation. However, the limited computation resources on clients’ devices prevent FL of large models. To overcome the constraint, one possible method is to reduce the computation memory usage with quantized neural networks such as quantization aware training on a centralized server. However, directly applying the quantization aware methods does not reduce the memory consumption on the clients’ devices of FL because the full-precision model is still used in the forward propagation of the model computation. To enable FL of the Conformer based ASR models, we propose FedAQT, an accurate quantized training framework under FL by training with quantized variables directly on clients’ devices. We empirically show that our method can achieve comparable WER with only 60% memory of the full-precision model.

Automatic speech recognition (ASR) models are typically trained on large datasets of transcribed speech. As language evolves and new terms come into use, these models can become outdated and stale. In the context of models trained on the server but deployed on edge devices, errors may result from the mismatch between server training data and actual on-device usage. In this work, we seek to continually learn from on-device user corrections through Federated Learning (FL) to address this issue. We explore techniques to target fresh terms that the model has not previously encountered, learn long-tail words, and mitigate catastrophic forgetting. In experimental evaluations, we find that the proposed techniques improve model recognition of fresh terms, while preserving quality on the overall language distribution.

This paper addresses the challenges of training large neural networks under federated learning settings: high on-device memory usage and communication cost. The proposed Online Model Compression (OMC) provides a framework that stores model parameters in a compressed format and decompresses them only when needed. We use quantization as the compression method in this paper and propose three methods, (1) per-variable transformation, (2) weight-matrix-only quantization, and (3) partial variable quantization, to minimize its impact on model accuracy. Our experiments on two recent neural networks for speech recognition and two different datasets show that OMC can reduce memory usage and communication cost of model parameters by up to 59% while attaining comparable accuracy and training speed when compared with full-precision federated learning.

Capitalization normalization (truecasing) is the task of restoring the correct case (uppercase or lowercase) of noisy text. We propose a fast, accurate and compact two-level hierarchical word-and-character-based recurrent neural network model. We use the truecaser to normalize user-generated text in a Federated Learning framework for language modeling. A case-aware language model trained on this normalized text achieves the same perplexity as a model trained on text with gold capitalization. In a real user A/B experiment, we demonstrate that the improvement translates to reduced prediction error rates in a virtual keyboard application. Similarly, in an ASR language model fusion experiment, we show reduction in uppercase character error rate and word error rate.

In distributed learning settings such as federated learning, the training algorithm can be potentially biased towards different clients.[1] proposed a domain-agnostic learning algorithm, where the model is optimized for any target distribution formed by a mixture of the client distributions in order to overcome this bias.They further proposed an algorithm for the cross-silo federated learning setting, where the number of clients is small.We consider this problem in the cross-device setting, where the number of clients is much larger.We propose a communicationefficient distributed algorithm called AGNOSTIC FEDERATED AVERAGING (or AGNOSTICFEDAVG) to minimize the domainagnostic objective proposed in [1], which is amenable to other private mechanisms such as secure aggregation.We highlight two types of naturally occurring domains in federated learning and argue that AGNOSTICFEDAVG performs well on both.To demonstrate the practical effectiveness of AGNOSTICFEDAVG, we report positive results for large-scale language modeling tasks in both simulation and live experiments, where the latter involves training language models for Spanish virtual keyboard for millions of user devices.

Mingqing Chen, Ananda Theertha Suresh, Rajiv Mathews, Adeline Wong, Cyril Allauzen, Françoise Beaufays, Michael Riley. Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL). 2019.

This paper presents a novel method for respiratory motion compensated reconstruction for cone beam computed tomography (CBCT). The reconstruction is based on a time sequence of motion vector fields, which is generated by a dynamic geometrical object shape model. The dynamic model is extracted from the 2D projection images of the CBCT. The process of the motion extraction is converted into an optimal 3D multiple interrelated surface detection problem, which can be solved by computing a maximum flow in a 4D directed graph. The method was tested on 12 mega-voltage (MV) CBCT scans from three patients. Two sets of motion-artifact-free 3D volumes, full exhale (FE) and full inhale (FI) phases, were reconstructed for each daily scan. The reconstruction was compared with three other motion-compensated approaches based on quantification accuracy of motion and size. Contrast-to-noise ratio (CNR) was also quantified for image quality. The proposed approach has the best overall performance, with a relative tumor volume quantification error of 3.39 ± 3.64% and 8.57 ± 8.31% for FE and FI phases, respectively. The CNR near the tumor area is 3.85 ± 0.42 (FE) and 3.58 ± 3.33 (FI). These results show the clinical feasibility to use the proposed method to reconstruct motion-artifact-free MVCBCT volumes.