Sequence-to-sequence (seq2seq) models achieve comparable or better grammatical error correction performance compared to sequenceto-edit (seq2edit) models.Seq2edit models normally iteratively refine the correction result, while seq2seq models decode only once without aware of subsequent tokens.Iteratively refining the correction results of seq2seq models via Multi-Pass Decoding (MPD) may lead to better performance.However, MPD increases the inference costs.Deleting or replacing corrections in previous rounds may lose useful information in the source input.We present an early-stop mechanism to alleviate the efficiency issue.To address the source information loss issue, we propose to merge the source input with the previous round correction result into one sequence.Experiments on the CoNLL-14 test set and BEA-19 test set show that our approach can lead to consistent and significant improvements over strong BART and T5 baselines (+1.80, +1.35, and +2.02 F0.5 for BART 12-2, large and T5 large respectively on CoNLL-14 and +2.99, +1.82, and +2.79 correspondingly on BEA-19), obtaining F0.5 scores of 68.41 and 75.36 on CoNLL-14 and BEA-19 respectively.We brought go to the orchard and apples but , forget pears .We bring go to the orchard and apples but , forget pears We bring go to the orchard and apples but , forget pears . .

Document-level neural machine translation (NMT) has outperformed sentence-level NMT on a number of datasets.However, documentlevel NMT is still not widely adopted in realworld translation systems mainly due to the lack of large-scale general-domain training data for document-level NMT.We examine the effectiveness of using Paracrawl for learning document-level translation.Paracrawl is a large-scale parallel corpus crawled from the Internet and contains data from various domains.The official Paracrawl corpus was released as parallel sentences (extracted from parallel webpages) and therefore previous works only used Paracrawl for learning sentence-level translation.In this work, we extract parallel paragraphs from Paracrawl parallel webpages using automatic sentence alignments and we use the extracted parallel paragraphs as parallel documents for training document-level translation models.We show that document-level NMT models trained with only parallel paragraphs from Paracrawl can be used to translate real documents from TED, News and Europarl, outperforming sentence-level NMT models.We also perform a targeted pronoun evaluation and show that document-level models trained with Paracrawl data can help context-aware pronoun translation.We release our data and code here 1 .

Jingyi Zhang, Josef van Genabith. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

The Transformer translation model (Vaswani et al., 2017) based on a multi-head attention mechanism can be computed effectively in parallel and has significantly pushed forward the performance of Neural Machine Translation (NMT).Though intuitively the attentional network can connect distant words via shorter network paths than RNNs, empirical analysis demonstrates that it still has difficulty in fully capturing long-distance dependencies (Tang et al., 2018).Considering that modeling phrases instead of words has significantly improved the Statistical Machine Translation (SMT) approach through the use of larger translation blocks ("phrases") and its reordering ability, modeling NMT at phrase level is an intuitive proposal to help the model capture long-distance relationships.In this paper, we first propose an attentive phrase representation generation mechanism which is able to generate phrase representations from corresponding token representations.In addition, we incorporate the generated phrase representations into the Transformer translation model to enhance its ability to capture long-distance relationships.In our experiments, we obtain significant improvements on the WMT 14 English-German and English-French tasks on top of the strong Transformer baseline, which shows the effectiveness of our approach.Our approach helps Transformer Base models perform at the level of Transformer Big models, and even significantly better for long sentences, but with substantially fewer parameters and training steps.The fact that phrase representations help even in the big setting further supports our conjecture that they make a valuable contribution to long-distance relations.

The Transformer translation model employs residual connection and layer normalization to ease the optimization difficulties caused by its multi-layer encoder/decoder structure. Previous research shows that even with residual connection and layer normalization, deep Transformers still have difficulty in training, and particularly Transformer models with more than 12 encoder/decoder layers fail to converge. In this paper, we first empirically demonstrate that a simple modification made in the official implementation, which changes the computation order of residual connection and layer normalization, can significantly ease the optimization of deep Transformers. We then compare the subtle differences in computation order in considerable detail, and present a parameter initialization method that leverages the Lipschitz constraint on the initialization of Transformer parameters that effectively ensures training convergence. In contrast to findings in previous research we further demonstrate that with Lipschitz parameter initialization, deep Transformers with the original computation order can converge, and obtain significant BLEU improvements with up to 24 layers. In contrast to previous research which focuses on deep encoders, our approach additionally enables Transformers to also benefit from deep decoders.

The translation quality estimation (QE) task, particularly the QE as a Metric task, aims to evaluate the general quality of a translation based on the translation and the source sentence without using reference translations.Supervised learning of this QE task requires human evaluation of translation quality as training data.Human evaluation of translation quality can be performed in different ways, including assigning an absolute score to a translation or ranking different translations.In order to make use of different types of human evaluation data for supervised learning, we present a multi-task learning QE model that jointly learns two tasks: score a translation and rank two translations.Our QE model exploits crosslingual sentence embeddings from pretrained multilingual language models.We obtain new state-of-the-art results on the WMT 2019 QE as a Metric task and outperform sentBLEU on the WMT 2019 Metrics task.