The rapid scaling of language models is motivating research using low-bitwidth quantization. In this work, we propose a novel binarization technique for Transformers applied to machine translation (BMT), the first of its kind. We identify and address the problem of inflated dot-product variance when using one-bit weights and activations. Specifically, BMT leverages additional LayerNorms and residual connections to improve binarization quality. Experiments on the WMT dataset show that a one-bit weight-only Transformer can achieve the same quality as a float one, while being 16$\times$ smaller in size. One-bit activations incur varying degrees of quality drop, but mitigated by the proposed architectural changes. We further conduct a scaling law study using production-scale translation datasets, which shows that one-bit weight Transformers scale and generalize well in both in-domain and out-of-domain settings. Implementation in JAX/Flax will be open sourced.

In this paper, we empirically study the optimization dynamics of multi-task learning, particularly focusing on those that govern a collection of tasks with significant data imbalance. We present a simple yet effective method of pre-training on high-resource tasks, followed by fine-tuning on a mixture of high/low-resource tasks. We provide a thorough empirical study and analysis of this method's benefits showing that it achieves consistent improvements relative to the performance trade-off profile of standard static weighting. We analyze under what data regimes this method is applicable and show its improvements empirically in neural machine translation (NMT) and multi-lingual language modeling.

In this work, we study the effect of varying the architecture and training data quality on the data scaling properties of Neural Machine Translation (NMT). First, we establish that the test loss of encoder-decoder transformer models scales as a power law in the number of training samples, with a dependence on the model size. Then, we systematically vary aspects of the training setup to understand how they impact the data scaling laws. In particular, we change the following (1) Architecture and task setup: We compare to a transformer-LSTM hybrid, and a decoder-only transformer with a language modeling loss (2) Noise level in the training distribution: We experiment with filtering, and adding iid synthetic noise. In all the above cases, we find that the data scaling exponents are minimally impacted, suggesting that marginally worse architectures or training data can be compensated for by adding more data. Lastly, we find that using back-translated data instead of parallel data, can significantly degrade the scaling exponent.

Recent research has proposed a series of specialized optimization algorithms for deep multi-task models. It is often claimed that these multi-task optimization (MTO) methods yield solutions that are superior to the ones found by simply optimizing a weighted average of the task losses. In this paper, we perform large-scale experiments on a variety of language and vision tasks to examine the empirical validity of these claims. We show that, despite the added design and computational complexity of these algorithms, MTO methods do not yield any performance improvements beyond what is achievable via traditional optimization approaches. We highlight alternative strategies that consistently yield improvements to the performance profile and point out common training pitfalls that might cause suboptimal results. Finally, we outline challenges in reliably evaluating the performance of MTO algorithms and discuss potential solutions.

We propose automatic speech recognition (ASR) models inspired by echo state network (ESN) [1], in which a subset of recurrent neural networks (RNN) layers in the models are randomly initialized and untrained. Our study focuses on RNN-T and Conformer models, and we show that model quality does not drop even when the decoder is fully randomized. Furthermore, such models can be trained more efficiently as the decoders do not require to be updated. By contrast, randomizing encoders hurts model quality, indicating that optimizing encoders and learn proper representations for acoustic inputs are more vital for speech recognition. Overall, we challenge the common practice of training ASR models for all components, and demonstrate that ESN-based models can perform equally well but enable more efficient training and storage than fully-trainable counterparts.

Owing to the features of erratic speeds and varying topography and requirements of minimum delay and high application reliability in terms of data delivery and security, existing MANET routing protocols prove to be inefficient in VANETs. In acknowledgment to the requirement for new VANET protocols addressing issues of routing, data dissemination, data sharing and security, this paper proposes a novel geographic routing technique called GROOV, which takes into account varying topographies and densities of highways as well as cities. To increase reliability, GROOV calculates transmission feasibility for each node, based on link quality (average acceleration), range weight (weightage to relative positions of nodes) and direction, instead of traditional greedy forwarding, in the selection of the next relay node. Taking volatility of critical city intersection scenarios into account, GROOV calculates new node coordinates of vehicles at intersections to make best route selections at intersections and thus, routes the data packet through the path directed at the intended recipient. This prevents the occurrence of a routing loop, thereby, decreasing delay and increasing packet delivery ratio. Simulation results show that GROOV achieves a high level of routing performance in terms of packet delivery ratio, end-to-end delay and average number of hops in both city straight road/highway and city intersection scenarios.

A user profile on an online social network is characterized by its profile entries (keywords). In this paper, we study the relationship between semantic similarity of user keywords and the social network topology. First, we present a 'forest' model to categorize keywords and define the notion of distance between keywords across multiple categorization trees (i.e., a forest). Second, we use the keyword distance to define similarity functions between a pair of users and show how social network topology can be modeled accordingly. Third, we validate our social network topology model, using a simulated social graph, against a real life social graph dataset.