The choice of tokenizer can profoundly impact language model performance, yet accessible and reliable evaluations of tokenizer quality remain an open challenge.Inspired by scaling consistency, we show that smaller models can accurately predict significant differences in tokenizer impact on larger models at a fraction of the compute cost.By systematically evaluating both English-centric and multilingual tokenizers, we find that tokenizer choice has negligible effects on tasks in English but results in consistent performance differences in multilingual settings.We propose new intrinsic tokenizer metrics inspired by Zipf's law that correlate more strongly with downstream performance than text compression when modeling unseen languages.By combining several metrics to capture multiple aspects of tokenizer behavior, we develop a reliable framework for intrinsic tokenizer evaluations.Our work offers a more efficient path to informed tokenizer selection in future language model development.

We propose utilizing n-best reranking to enhance Sequence-Level Knowledge Distillation (Kim and Rush, 2016) where we extract pseudolabels for student model's training data from top n-best hypotheses and leverage a diverse set of models with different inductive biases, objective functions or architectures, including some publicly-available large language models, to pick the highest-quality hypotheses as labels.The effectiveness of our proposal is validated through experiments on the WMT'21 German ↔ English and Chinese ↔ English translation tasks.Our results demonstrate that utilizing pseudo-labels generated by our n-best reranker leads to a significantly more accurate student model.In fact, our best student model achieves comparable accuracy to a large translation model from (Tran et al., 2021) with 4.7 billion parameters, while having two orders of magnitude fewer parameters.

Variational Neural Machine Translation (VNMT) is an attractive framework for modeling the generation of target translations, conditioned not only on the source sentence but also on some latent random variables.The latent variable modeling may introduce useful statistical dependencies that can improve translation accuracy.Unfortunately, learning informative latent variables is non-trivial, as the latent space can be prohibitively large, and the latent codes are prone to be ignored by many translation models at training time.Previous works impose strong assumptions on the distribution of the latent code and limit the choice of the NMT architecture.In this paper, we propose to apply the VNMT framework to the state-of-the-art Transformer and introduce a more flexible approximate posterior based on normalizing flows.We demonstrate the efficacy of our proposal under both in-domain and out-of-domain conditions, significantly outperforming strong baselines.

We present a three-pronged approach to improving Statistical Machine Translation (SMT), building on recent success in the application of neural networks to SMT. First, we propose new features based on neural networks to model various non-local translation phenomena. Second, we augment the architecture of the neural network with tensor layers that capture important higher-order interaction among the network units. Third, we apply multitask learning to estimate the neural network parameters jointly. Each of our proposed methods results in significant improvements that are complementary. The overall improvement is +2.7 and +1.8 BLEU points for Arabic-English and Chinese-English translation over a state-of-the-art system that already includes neural network features.

Reordering poses one of the greatest challenges in Statistical Machine Translation research as the key contextual information may well be beyond the confine of translation units.We present the "Anchor Graph" (AG) model where we use a graph structure to model global contextual information that is crucial for reordering.The key ingredient of our AG model is the edges that capture the relationship between the reordering around a set of selected translation units, which we refer to as anchors.As the edges link anchors that may span multiple translation units at decoding time, our AG model effectively encodes global contextual information that is previously absent.We integrate our proposed model into a state-of-the-art translation system and demonstrate the efficacy of our proposal in a largescale Chinese-to-English translation task.* This work was done when the authors were with IBM. 1 We define translation units as phrases in phrase-based SMT or as translation rules in syntax-based SMT.

Long Term Evolution (LTE) physical layer identity can be recognized either in frequency domain or time domain. In this paper we serve some comparisons in term of complexity, latency, and recognition probability. As a result, frequency domain detection has superiority on performance for SNR higher than 3dB on all channel models, as well as lower latency than time domain scheme one. However, it requires huge computation resources that might meet difficulty for mobile devices implementation.

This paper discusses about RTL design of time domain auto-correlation architecture for two recent standards, i.e.Worldwide Interoperability for Microwave Access Orthogonal Frequency Division Multiple Access (WiMAX OFDMA) and 3rd Generation Partnership Project Long Term Evolution (3GPP LTE). We introduce time domain auto-correlation function to recognize WiMAX preamble and LTE primary synchronization channel (PSC) at the same time. To distinguish them from interference, we consider peak period as unique property of WiMAX's preamble and LTE's PSC. As a result, for SNR 0 dB in extended pedestrian A (EPA) and ITU pedestrian B channel models, more than 90% WiMAX's preamble as well as LTE's PSC can be detected correctly. From complexity point of view, our scheme requires 183k LUT when implemented into Xilinx4 xc4vlx200.

We propose a simplified Level Of Detail (LOD) algorithm to learn phrase translation for statistical machine translation. In particular, LOD learns unknown phrase translations from parallel texts without linguistic knowledge. LOD uses an agglomerative method to attack the combinatorial explosion that results when generating candidate phrase translations. Although LOD was previously proposed by (Setiawan et al., 2005), we improve the original algorithm in two ways: simplifying the algorithm and using a simpler translation model. Experimental results show that our algorithm provides comparable performance while demonstrating a significant reduction in computation time.