Deep neural networks have recently become the gold standard for acoustic modeling in speech recognition systems. The key computational unit of a deep network is a linear projection followed by a point-wise non-linearity, which is typically a logistic function. In this work, we show that we can improve generalization and make training of deep networks faster and simpler by substituting the logistic units with rectified linear units. These units are linear when their input is positive and zero otherwise. In a supervised setting, we can successfully train very deep nets from random initialization on a large vocabulary speech recognition task achieving lower word error rates than using a logistic network with the same topology. Similarly in an unsupervised setting, we show how we can learn sparse features that can be useful for discriminative tasks. All our experiments are executed in a distributed environment using several hundred machines and several hundred hours of speech data.

Recent work in unsupervised feature learning and deep learning has shown that being able to train large models can dramatically improve performance. In this paper, we consider the problem of training a deep network with billions of parameters using tens of thousands of CPU cores. We have developed a software framework called DistBelief that can utilize computing clusters with thousands of machines to train large models. Within this framework, we have developed two algorithms for large-scale distributed training: (i) Downpour SGD, an asynchronous stochastic gradient descent procedure supporting a large number of model replicas, and (ii) Sandblaster, a framework that supports for a variety of distributed batch optimization procedures, including a distributed implementation of L-BFGS. Downpour SGD and Sandblaster L-BFGS both increase the scale and speed of deep network training. We have successfully used our system to train a deep network 100x larger than previously reported in the literature, and achieves state-of-the-art performance on ImageNet, a visual object recognition task with 16 million images and 21k categories. We show that these same techniques dramatically accelerate the training of a more modestly sized deep network for a commercial speech recognition service. Although we focus on and report performance of these methods as applied to training large neural networks, the underlying algorithms are applicable to any gradient-based machine learning algorithm.

A vector extension of a necessary condition for asymptotically optimal stationary (sliding-block) source codes is presented. The condition implies the intuitive result that the reproduction process for an IID input must be approximately uncorrelated if the code is approximately optimal, a property previously demonstrated empirically for common examples. One-bit coding of a Gaussian IID process is used to illustrate the goodness of fit of the empirical distribution of the reproduction to the Shannon optimal distribution.

Necessary conditions for asymptotically optimal sliding-block or stationary codes for source coding and rate-constrained simulation of memoryless sources are presented and used to motivate a design technique for trellis-encoded source coding and rate-constrained simulation. The code structure has intuitive similarities to classic random coding arguments as well as to “fake process” methods and alphabet-constrained methods. Experimental evidence shows that the approach provides comparable or superior performance in comparison with previously published methods on common examples, sometimes by significant margins.

Necessary conditions for asymptotically optimal sliding-block or stationary codes for source coding and rate-constrained simulation are presented and applied to a design technique for trellis-encoded source coding and rate constrained simulation of memoryless sources.

We propose a new real-time packet scheduling algorithm for streaming scalable H.264. Our algorithm makes use of a packet importance measure, which we define, that takes into consideration transmission history, channel conditions, and the unique decoding dependencies due to the temporal wavelet encoding. Our algorithm utilizes this importance measure to minimize the expected reconstruction distortion at the decoder under a certain rate constraint. In our experimental results we show gains of more than 3 dB in decoded video quality when transmissions are controlled with our algorithm as compared to existing schedulers

A common approach to automatic speech recognition uses two recognition passes to decode an utterance: the first pass limits the search to a smaller set of likely hypotheses; the second pass re-scores the limited set using more detailed acoustic models which may target gender or specific channels. A question raised by this architecture is how to define and train the second pass models. We describe an extensible automatic solution that requires no manual gender or channel labeling. To train the second pass models, we cluster the training data into datasets containing utterances whose acoustics are very similar across the entire utterance. The clustering is based on which regions of a more general acoustic model are activated during forced alignments. Experiments with commercial American-English digit strings show 9.3% relative error rate reduction over a gender-based two pass system with similar numbers of model parameters.

This paper presents a novel Bayesian approach to the problem of co-channel speech. The problem is for-mulated as the joint maximization of the a posteriori probability of the word sequence and the target speaker given the observed speech signal. It is shown that the joint probability can be expressed as the product of six terms: a likelihood score from a speaker-independent speech recognizer, the (normalized) likelihood score of a speaker recognizer, the likelihood of a sequence of prosodic events, the likelihood of a speaker-dependent statistical language model, a prior representing the chan-nel usage patterns of a speaker, and the prior probabil-ity of the speaker. An efficient single-pass Viterbi search strategy is presented. Experimental results on over-the-telephone recognition of co-channel speech show a 45% reduction in word error rate of a 10-digit telephone num-ber task. 1.

We propose a new approach for designing compact acoustic models particularly suited to large systems that combine multiple model sets to represent distinct acoustic conditions or languages. We show that Gaussians based on mixtures of inverse covariances (MIC) with shared parameters can be clustered using an efficient Lloyd algorithm. As a result, more compact acoustic models can be built by clustering Gaussians across tied mixtures. In addition, we show that the tied parameters of MIC models can be shared across acoustic models and languages, making it possible to build more efficient multi-model systems which take advantage of a common pool of clustered Gaussians. 1.