Neural Network Language Models (NNLMs) for Virtual Assistants (VAs) are generally language-, region-, and in some cases, device-dependent, which increases the effort to scale and maintain them. Combining NNLMs for one or more of the categories is one way to improve scalability. In this work, we combine regional variants of English to build a "World English" NNLM for on-device VAs. In particular, we investigate the application of adapter bottlenecks to model dialect-specific characteristics in our existing production NNLMs and enhance the multi-dialect baselines. We find that adapter modules are more effective in modeling dialects than specializing entire sub-networks. Based on this insight and leveraging the design of our production models, we introduce a new architecture for World English NNLM that meets the accuracy, latency and memory constraints of our single-dialect models.

Federated Learning (FL) is a technique to train models on distributed edge devices with local data samples. Differential Privacy (DP) can be applied with FL to provide a formal privacy guarantee for sensitive data on device. Our goal is to train a large neural network language model (NNLM) on compute-constrained devices while preserving privacy using FL and DP. However, the noise required to guarantee differential privacy increases as the model size grows, which often prevents convergence. We propose Partial Embedding Updates (PEU), a novel technique to reduce the impact of DP-noise by decreasing payload size. Furthermore, we adopt Low Rank Adaptation (LoRA) and Noise Contrastive Estimation (NCE) to reduce the memory demands of large models on compute-constrained devices. We demonstrate in simulation and with real devices that this combination of techniques makes it possible to train large-vocabulary language models while preserving accuracy and privacy.

Graphics Processing Units (GPUs) are commonly used to train and evaluate neural networks efficiently.While previous work in deep learning has focused on accelerating operations on dense matrices/tensors on GPUs, efforts have concentrated on operations involving sparse data structures.Operations using sparse structures are common in natural language models at the input and output layers, because these models operate on sequences over discrete alphabets.We present two new GPU algorithms: one at the input layer, for multiplying a matrix by a few-hot vector (generalizing the more common operation of multiplication by a one-hot vector) and one at the output layer, for a fused softmax and top-N selection (commonly used in beam search).Our methods achieve speedups over state-of-theart parallel GPU baselines of up to 7× and 50×, respectively.We also illustrate how our methods scale on different GPU architectures.

Weighted finite state transducers (FSTs) are frequently used in language processing to handle tasks such as part-of-speech tagging and speech recognition.There has been previous work using multiple CPU cores to accelerate finite state algorithms, but limited attention has been given to parallel graphics processing unit (GPU) implementations.In this paper, we introduce the first (to our knowledge) GPU implementation of the FST composition operation, and we also discuss the optimizations used to achieve the best performance on this architecture.We show that our approach obtains speedups of up to 6× over our serial implementation and 4.5× over OpenFST.

Weighted finite automata and transducers (including hidden Markov models and conditional random fields) are widely used in natural language processing (NLP) to perform tasks such as morphological analysis, part-of-speech tagging, chunking, named entity recognition, speech recognition, and others.Parallelizing finite state algorithms on graphics processing units (GPUs) would benefit many areas of NLP.Although researchers have implemented GPU versions of basic graph algorithms, limited previous work, to our knowledge, has been done on GPU algorithms for weighted finite automata.We introduce a GPU implementation of the Viterbi and forward-backward algorithm, achieving decoding speedups of up to 5.2x over our serial implementation running on different computer architectures and 6093x over OpenFST.

Social sensing has emerged as a new application paradigm in networked sensing communities where a colossal amount of observations about the physical world are contributed by people or devices they use. Our work solves a critical challenge in social sensing applications where the goal is to estimate the reliability of social sensors and the truthfulness of observed variables (typically known as claims) with little prior knowledge on either of them. This challenge is referred to as truth discovery. An important limitation in the previous truth discovery solutions is that they assume the relationship between source reliability and claim truthfulness can be represented by simplified functions (e.g., linear, quadratic and binomial). This assumption leads to suboptimal truth discovery results because the exact relational dependency between sources and claims is often unknown a priori. In this paper, we show that a neural network approach can learn the complex relational dependency better than the previous truth discovery methods. In particular, we develop a multi-layer neural network model that solves the truth discovery problem in social sensing without any assumption on the prior knowledge of the source-claim relational dependency distribution. The performance of our model is evaluated through two real-world events using data crawled from Twitter. The evaluation results show that our neural network approach significantly outperforms previous truth discovery methods.