Neural Network Language Models (NNLMs) for Virtual Assistants (VAs) are generally language-, region-, and in some cases, device-dependent, which increases the effort to scale and maintain them. Combining NNLMs for one or more of the categories is one way to improve scalability. In this work, we combine regional variants of English to build a "World English" NNLM for on-device VAs. In particular, we investigate the application of adapter bottlenecks to model dialect-specific characteristics in our existing production NNLMs and enhance the multi-dialect baselines. We find that adapter modules are more effective in modeling dialects than specializing entire sub-networks. Based on this insight and leveraging the design of our production models, we introduce a new architecture for World English NNLM that meets the accuracy, latency and memory constraints of our single-dialect models.

Virtual Assistants (VAs) are important Information Retrieval platforms that help users accomplish various tasks through spoken commands. The speech recognition system (speech-to-text) uses query priors, trained solely on text, to distinguish between phonetically confusing alternatives. Hence, the generation of synthetic queries that are similar to existing VA usage can greatly improve upon the VA's abilities---especially for use-cases that do not (yet) occur in paired audio/text data. In this paper, we provide a preliminary exploration of the use of Large Language Models (LLMs) to generate synthetic queries that are complementary to template-based methods. We investigate whether the methods (a) generate queries that are similar to randomly sampled, representative, and anonymized user queries from a popular VA, and (b) whether the generated queries are specific. We find that LLMs generate more verbose queries, compared to template-based methods, and reference aspects specific to the entity. However, the generated queries are similar to VA user queries, and are specific enough to retrieve the relevant entity. We conclude that queries generated by LLMs and templates are complementary.

Virtual assistants make use of automatic speech recognition (ASR) to help users answer entity-centric queries.However, spoken entity recognition is a difficult problem, due to the large number of frequently-changing named entities.In addition, resources available for recognition are constrained when ASR is performed on-device.In this work, we investigate the use of probabilistic grammars as language models within the finite-state transducer (FST) framework.We introduce a deterministic approximation to probabilistic grammars that avoids the explicit expansion of non-terminals at model creation time, integrates directly with the FST framework, and is complementary to n-gram models.We obtain a 10% relative word error rate improvement on long tail entity queries compared to when a similarly-sized n-gram model is used without our method.

In this work, we uncover a theoretical connection between two language model interpolation techniques, count merging and Bayesian interpolation. We compare these techniques as well as linear interpolation in three scenarios with abundant training data per component model. Consistent with prior work, we show that both count merging and Bayesian interpolation outperform linear interpolation. We include the first (to our knowledge) published comparison of count merging and Bayesian interpolation, showing that the two techniques perform similarly. Finally, we argue that other considerations will make Bayesian interpolation the preferred approach in most circumstances.

Automatic Speech Recognition (ASR) has recently proved to \nbe a useful tool to reduce the workload of air traffic controllers \nleading to significant gains in operational efficiency. Air Traffic Control (ATC) systems in operation rooms around the world \ngenerate large amounts of untranscribed speech and radar data \neach day, which can be utilized to build and improve ASR models. In this paper, we propose an iterative approach that utilizes \nincreasing amounts of untranscribed data to incrementally build \nthe necessary ASR models for an ATC operational area. Our approach uses a semi-supervised learning framework to combine \nspeech and radar data to iteratively update the acoustic model, \nlanguage model and command prediction model (i.e. prediction \nof possible commands from radar data for a given air traffic \nsituation) of an ASR system. Starting with seed models built \nwith a limited amount of manually transcribed data, we simulate an operational scenario to adapt and improve the models \nthrough semi-supervised learning. Experiments on two independent ATC areas (Vienna and Prague) demonstrate the utility \nof our proposed methodology that can scale to operational environments with minimal manual effort for learning and adaptation.

Automatic Speech Recognition and Understanding (ASRU) systems can generally use temporal and situational context information to improve their performance for a given task. This is typically done by rescoring the ASR hypotheses or by dynamically adapting the ASR models. For some domains, such as Air Traffic Control (ATC), this context information can be, however, small in size, partial and available only as abstract concepts (e.g. airline codes), which are difficult to map into full possible spoken sentences to perform rescoring or adaptation. This paper presents a multi-modal ASRU system, which dynamically integrates partial temporal and situational ATC context information to improve its performance. This is done either by 1) extracting word sequences which carry relevant ATC information from ASR N-best Lists and then perform a context-based rescoring on the extracted ATC segments or 2) by a partial adaptation of the language model. Experiments conducted on 4 hours of test data from Prague and Vienna approach (arrivals) showed a relative reduction of the ATC command error rate metric by 30% to 50%.

The performance of Neural Network (NN)-based language models is steadily improving due to the emergence of new architectures, which are able to learn different natural language characteristics. This paper presents a novel framework, which shows that a significant improvement can be achieved by combining different existing heterogeneous models in a single architecture. This is done through 1) a feature layer, which separately learns different NN-based models and 2) a mixture layer, which merges the resulting model features. In doing so, this architecture benefits from the learning capabilities of each model with no noticeable increase in the number of model parameters or the training time. Extensive experiments conducted on the Penn Treebank (PTB) and the Large Text Compression Benchmark (LTCB) corpus showed a significant reduction of the perplexity when compared to state-of-the-art feedforward as well as recurrent neural network architectures.

Training large vocabulary Neural Network Language Models (NNLMs) is a difficult task due to the explicit requirement of the output layer normalization, which typically involves the evaluation of the full softmax function over the complete vocabulary. This paper proposes a Batch Noise Contrastive Estimation (B-NCE) approach to alleviate this problem. This is achieved by reducing the vocabulary, at each time step, to the target words in the batch and then replacing the softmax by the noise contrastive estimation approach, where these words play the role of targets and noise samples at the same time. In doing so, the proposed approach can be fully formulated and implemented using optimal dense matrix operations. Applying B-NCE to train different NNLMs on the Large Text Compression Benchmark (LTCB) and the One Billion Word Benchmark (OBWB) shows a significant reduction of the training time with no noticeable degradation of the models performance. This paper also presents a new baseline comparative study of different standard NNLMs on the large OBWB on a single Titan-X GPU.

Automatic Speech Recognition (ASR) can introduce higher levels \nof automation into Air Traffic Control (ATC), where spoken \nlanguage is still the predominant form of communication. \nWhile ATC uses standard phraseology and a limited vocabulary, \nwe need to adapt the speech recognition systems to local \nacoustic conditions and vocabularies at each airport to reach \noptimal performance. Due to continuous operation of ATC systems, \na large and increasing amount of untranscribed speech \ndata is available, allowing for semi-supervised learning methods \nto build and adapt ASR models. In this paper, we first identify \nthe challenges in building ASR systems for specific ATC \nareas and propose to utilize out-of-domain data to build baseline \nASR models. Then we explore different methods of data \nselection for adapting baseline models by exploiting the continuously \nincreasing untranscribed data. We develop a basic approach \ncapable of exploiting semantic representations of ATC \ncommands. We achieve relative improvement in both word error \nrate (23.5%) and concept error rates (7%) when adapting \nASR models to different ATC conditions in a semi-supervised \nmanner.

Secure user identification is important for the increasing number of eyewear computers but limited input capabilities pose significant usability challenges for established knowledge-based schemes, such as passwords or PINs. We present SkullConduct, a biometric system that uses bone conduction of sound through the user's skull as well as a microphone readily integrated into many of these devices, such as Google Glass. At the core of SkullConduct is a method to analyze the characteristic frequency response created by the user's skull using a combination of Mel Frequency Cepstral Coefficient (MFCC) features as well as a computationally light-weight 1NN classifier. We report on a controlled experiment with 10 participants that shows that this frequency response is person-specific and stable -- even when taking off and putting on the device multiple times -- and thus serves as a robust biometric. We show that our method can identify users with 97.0% accuracy and authenticate them with an equal error rate of 6.9%, thereby bringing biometric user identification to eyewear computers equipped with bone conduction technology.

Training good word embeddings requires large amounts of data. Out-of-vocabulary words will still be encountered at test-time, leaving these words without embeddings. To overcome this lack of embeddings for rare words, existing methods leverage morphological features to generate embeddings. While the existing methods use computationally-intensive rule-based (Soricut and Och, 2015) or tool-based (Botha and Blunsom, 2014) morphological analysis to generate embeddings, our system applies a computationally-simpler sub-word search on words that have existing embeddings. Embeddings of the sub-word search results are then combined using string similarity functions to generate rare word embeddings. We augmented pre-trained word embeddings with these novel embeddings and evaluated on a rare word similarity task, obtaining up to 3 times improvement in correlation over the original set of embeddings. Applying our technique to embeddings trained on larger datasets led to on-par performance with the existing state-of-the-art for this task. Additionally, while analysing augmented embeddings in a log-bilinear language model, we observed up to 50% reduction in rare word perplexity in comparison to other more complex language models.

The goal of language modeling techniques is to capture the statistical and structural properties of natural languages from training corpora.This task typically involves the learning of short range dependencies, which generally model the syntactic properties of a language and/or long range dependencies, which are semantic in nature.We propose in this paper a new multi-span architecture, which separately models the short and long context information while it dynamically merges them to perform the language modeling task.This is done through a novel recurrent Long-Short Range Context (LSRC) network, which explicitly models the local (short) and global (long) context using two separate hidden states that evolve in time.This new architecture is an adaptation of the Long-Short Term Memory network (LSTM) to take into account the linguistic properties.Extensive experiments conducted on the Penn Treebank (PTB) and the Large Text Compression Benchmark (LTCB) corpus showed a significant reduction of the perplexity when compared to state-of-the-art language modeling techniques.

Feedforward Neural Network (FNN)-based language models estimate the probability of the next word based on the history of the last N words, whereas Recurrent Neural Networks (RNN) perform the same task based only on the last word and some context information that cycles in the network. This paper presents a novel approach, which bridges the gap between these two categories of networks. In particular, we propose an architecture which takes advantage of the explicit, sequential enumeration of the word history in FNN structure while enhancing each word representation at the projection layer through recurrent context information that evolves in the network. The context integration is performed using an additional word-dependent weight matrix that is also learned during the training. Extensive experiments conducted on the Penn Treebank (PTB) and the Large Text Compression Benchmark (LTCB) corpus showed a significant reduction of the perplexity when compared to state-of-the-art feedforward as well as recurrent neural network architectures.

The use of prior situational/contextual knowledge about a given \ntask can significantly improve automatic speech recognition \n(ASR) performance. This is typically done through adaptation \nof acoustic or language models if data is available or using \nknowledge-based rescoring. The main adaptation techniques, \nhowever, are either domain-specific, which makes them inadequate \nfor other tasks, or static and offline, and therefore cannot \ndeal with dynamic knowledge. To circumvent this problem, \nwe propose a real-time system which dynamically integrates \nsituational context into ASR. The context integration is done \neither post-recognition, in which case a weighted Levenshtein \ndistance between the ASR hypotheses and the context information \nbased on the ASR confidence scores is proposed to extract \nthe most likely sequence of spoken words, or pre-recognition, \nwhere the search space is adjusted to the new situational knowledge \nthrough adaptation of the finite state machine modeling \nthe spoken language. Experiments conducted on 3 hours of \nAir Traffic Control (ATC) data achieved a 51% reduction of \nthe Command Error Rate (CmdER) which is used as evaluation \nmetric in the ATC domain.

This paper presents a novel filtering approach for tracking multiple concurrent speakers with a microphone array. In this framework, a Kalman filter bank that evolves in time according to a temporal Hidden Markov Model (HMM) is proposed. This approach was designed to overcome two major problems that occur in spontaneous speech; namely, 1) the speaker overlap. This problem is solved using a bank of parallel Kalman filters that track multiple simultaneous speakers, and 2) the high discontinuity of spontaneous speech caused by short breaks and silences. This is solved using an HMM that allows speakers to change their state (speaking, silent, etc.) over time. The actual active speakers number and locations are extracted from the active filters using a second Kalman filter. Experiments on the AV16.3 showed an average tracking rate improvement of 8% compared to a short-term clustering approach, while being 7 times faster.

This paper presents an approach for incorporating situational context information into an on-line Automatic Speech Recognition (ASR) component of an Air Traffic Control (ATC) assistance system to improve recognition performance. Context information is treated as prior information to reduce the search space for recognition. It is integrated in the ASR pipeline by continually updating the recognition network. This is achieved by automatically adapting the underlying grammar whenever new situational knowledge becomes available. The context-dependent recognition network is then re-created and substituted for recognition based on these context-dependent grammars. As a result, the recognizer's search space is constantly being limited to that subset of hypotheses that are deemed plausible in the current situation. Since recognition and adaptation tasks can be easily performed by two separate parallel processes, on-line capabilities of the system are maintained, and response times do not increase as a result of context integration. Experiments conducted on about two hours of ATC data show a reduction in command error rate by a factor of three when context is used.

Adaptation techniques for speech recognition are very effective in single-speaker scenarios. However, when distant microphones capture overlapping speech from multiple speakers, conventional speaker adaptation methods are less effective. The putative signal for any speaker contains interference from other speakers. Consequently, any adaptation technique adapts the model to the interfering speakers as well, which leads to degradation of recognition performance for the desired speaker. In this work, we develop a new feature-space adaptation method for overlapping speech. We first build a beamformer to enhance speech from each active speaker. After that, we compute speech feature vectors from the output of each beamformer. We then jointly transform the feature vectors from all speakers to maximize the likelihood of their respective acoustic models. Experiments run on the speech separation challenge data collected under the AMI project demonstrate the effectiveness of our adaptation method. An absolute word error rate (WER) reduction up to 14 % was achieved in the case of delay-and-sum beamforming. With minimum mutual information (MMI) beamforming, our adaptation method achieved a WER of 31.5 %. To the best of our knowledge, this is the lowest WER reported on this task.

This paper presents a novel probabilistic framework for localizing multiple speakers with a microphone array. In this framework, the generalized cross correlation function (GCC) of each microphone pair is interpreted as a probability distribution of the time difference of arrival (TDOA) and subsequently approximated as a Gaussian mixture. The distribution parameters are estimated with a weighted expectation maximization algorithm. Then, the joint distribution of the TDOA Gaussian mixtures is mapped to a multimodal distribution in the location space, where each mode represents a potential source location. The approach taken here performs the localization by 1) reducing the search space to some regions that are likely to contain a source and then 2) extracting the actual speaker locations with a numerical optimization algorithm. The effectiveness of the proposed approach is shown using the AV16.3 corpus.

Multiple speaker localization algorithms generally require a bi-nary detector, which performs the source/noise classification of the location estimates. This is mainly due to the unknown time-varying number of sources, and to the presence of noise and reverberation. In this paper, we propose an unsupervised learn-ing approach based on a naive Bayesian classifier. The proposed approach couples two speaker location features, namely, 1) the steered response power introduced at the location estimate, and 2) the corresponding maximum likelihood error, which charac-terizes the variance of the estimate. The latter is experimentally shown to be highly correlated with the steered power at the loca-tion estimate. The proposed method is further extended to con-trol the misclassification rate through the use of a loss function. This approach is general, and can be easily extended to integrate more speaker/speech features. Experiments on the AV16.3 cor-pus show the effectiveness of the proposed approach. Index Terms: microphone arrays, multiple speaker localiza-tion, source detection, Bayesian classification.