Neural Machine Translation (NMT) models can be specialized by domain adaptation, often involving fine-tuning on a dataset of interest.This process risks catastrophic forgetting: rapid loss of generic translation quality.Forgetting has been widely observed, with many mitigation methods proposed.However, the causes of forgetting and the relationship between forgetting and adaptation data are under-explored.This paper takes a novel approach to understanding catastrophic forgetting during NMT adaptation by investigating the impact of the data.We provide a first investigation of what is forgotten, and why.We examine the relationship between forgetting and the in-domain data, and show that the amount and type of forgetting is linked to that data's target vocabulary coverage.Our findings pave the way toward better informed NMT domain adaptation.

The development of deep learning techniques has allowed Neural Machine Translation (NMT) models to become extremely powerful, given sufficient training data and training time. However, systems struggle when translating text from a new domain with a distinct style or vocabulary. Fine-tuning on in-domain data allows good domain adaptation, but requires sufficient relevant bilingual data. Even if this is available, simple fine-tuning can cause overfitting to new data and catastrophic forgetting of previously learned behaviour. We survey approaches to domain adaptation for NMT, particularly where a system may need to translate across multiple domains. We divide techniques into those revolving around data selection or generation, model architecture, parameter adaptation procedure, and inference procedure. We finally highlight the benefits of domain adaptation and multidomain adaptation techniques to other lines of NMT research.

Training data for NLP tasks often exhibits gender bias in that fewer sentences refer to women than to men.In Neural Machine Translation (NMT) gender bias has been shown to reduce translation quality, particularly when the target language has grammatical gender.The recent WinoMT challenge set allows us to measure this effect directly (Stanovsky et al., 2019).Ideally we would reduce system bias by simply debiasing all data prior to training, but achieving this effectively is itself a challenge.Rather than attempt to create a 'balanced' dataset, we use transfer learning on a small set of trusted, gender-balanced examples.This approach gives strong and consistent improvements in gender debiasing with much less computational cost than training from scratch.A known pitfall of transfer learning on new domains is 'catastrophic forgetting', which we address both in adaptation and in inference. During adaptation we show that Elastic Weight Consolidation allows a performance trade-off between general translation quality and bias reduction. During inference we propose a latticerescoring scheme which outperforms all systems evaluated in Stanovsky et al. (2019) onWinoMT with no degradation of general test set BLEU, and we show this scheme can be applied to remove gender bias in the output of 'black box' online commercial MT systems.We demonstrate our approach translating from English into three languages with varied linguistic properties and data availability.

We present Neural Machine Translation (NMT) training using document-level metrics with batch-level documents.Previous sequence-objective approaches to NMT training focus exclusively on sentence-level metrics like sentence BLEU which do not correspond to the desired evaluation metric, typically document BLEU.Meanwhile research into document-level NMT training focuses on data or model architecture rather than training procedure.We find that each of these lines of research has a clear space in it for the other, and propose merging them with a scheme that allows a document-level evaluation metric to be used in the NMT training objective.We first sample pseudo-documents from sentence samples.We then approximate the expected document BLEU gradient with Monte Carlo sampling for use as a cost function in Minimum Risk Training (MRT).This twolevel sampling procedure gives NMT performance gains over sequence MRT and maximum-likelihood training.We demonstrate that training is more robust for document-level metrics than with sequence metrics.We further demonstrate improvements on NMT with TER and Grammatical Error Correction (GEC) using GLEU, both metrics used at the document level for evaluations.

We investigate adaptive ensemble weighting for Neural Machine Translation, addressing the case of improving performance on a new and potentially unknown domain without sacrificing performance on the original domain.We adapt sequentially across two Spanish-English and three English-German tasks, comparing unregularized fine-tuning, L2 and Elastic Weight Consolidation.We then report a novel scheme for adaptive NMT ensemble decoding by extending Bayesian Interpolation with source information, and show strong improvements across test domains without access to the domain label.

Word units are a popular choice in statistical language modelling. For inflective and agglutinative languages this choice may result in a high out of vocabulary rate. Subword units, such as morphs, provide an interesting alternative to words. These units can be derived in an unsupervised fashion and empirically show lower out of vocabulary rates. This paper proposes a morph-to-word transduction to convert morph sequences into word sequences. This enables powerful word language models to be applied. In addition, it is expected that techniques such as pruning, confusion network decoding, keyword search and many others may benefit from word rather than morph level decision making. However, word or morph systems alone may not achieve optimal performance in tasks such as keyword search so a combination is typically employed. This paper proposes a single index approach that enables word, morph and phone searches to be performed over a single morph index. Experiments are conducted on IARPA Babel program languages including the surprise languages of the OpenKWS 2015 and 2016 competitions.