We propose a novel data-augmentation technique for neural machine translation based on ROT-k ciphertexts.ROT-k is a simple letter substitution cipher that replaces a letter in the plaintext with the kth letter after it in the alphabet.We first generate multiple ROT-k ciphertexts using different values of k for the plaintext which is the source side of the parallel data.We then leverage this enciphered training data along with the original parallel data via multi-source training to improve neural machine translation.Our method, CipherDAug, uses a co-regularization-inspired training procedure, requires no external data sources other than the original training data, and uses a standard Transformer to outperform strong data augmentation techniques on several datasets by a significant margin.This technique combines easily with existing approaches to data augmentation, and yields particularly strong results in low-resource settings.1

We propose a novel technique that combines alternative subword tokenizations of a single source-target language pair that allows us to leverage multilingual neural translation training methods. These alternate segmentations function like related languages in multilingual translation. Overall this improves translation accuracy for low-resource languages and produces translations that are lexically diverse and morphologically rich. We also introduce a cross-teaching technique which yields further improvements in translation accuracy and cross-lingual transfer between high- and low-resource language pairs. Compared to other strong multilingual baselines, our approach yields average gains of +1.7 BLEU across the four low-resource datasets from the multilingual TED-talks dataset. Our technique does not require additional training data and is a drop-in improvement for any existing neural translation system.

While the attention heatmaps produced by neural machine translation (NMT) models seem insightful, there is little evidence that they reflect a model's true internal reasoning.We provide a measure of faithfulness for NMT based on a variety of stress tests where attention weights which are crucial for prediction are perturbed and the model should alter its predictions if the learned weights are a faithful explanation of the predictions.We show that our proposed faithfulness measure for NMT models can be improved using a novel differentiable objective that rewards faithful behaviour by the model through probability divergence.Our experimental results on multiple language pairs show that our objective function is effective in increasing faithfulness and can lead to a useful analysis of NMT model behaviour and more trustworthy attention heatmaps.Our proposed objective improves faithfulness without reducing the translation quality and has a useful regularization effect on the NMT model and can even improve translation quality in some cases.

Decipherment of homophonic substitution ciphers using language models (LMs) is a wellstudied task in NLP.Previous work in this topic scores short local spans of possible plaintext decipherments using n-gram LMs.The most widely used technique is the use of beam search with n-gram LMs proposed by Nuhn et al. (2013).We propose a beam search algorithm that scores the entire candidate plaintext at each step of the decipherment using a neural LM.We augment beam search with a novel rest cost estimation that exploits the prediction power of a neural LM.We compare against the state of the art n-gram based methods on many different decipherment tasks.On challenging ciphers such as the Beale cipher we provide significantly better error rates with much smaller beam sizes.