Automatic metrics are fundamental for the development and evaluation of machine translation systems.Judging whether, and to what extent, automatic metrics concur with the gold standard of human evaluation is not a straightforward problem.We show that current methods for judging metrics are highly sensitive to the translations used for assessment, particularly the presence of outliers, which often leads to falsely confident conclusions about a metric's efficacy.Finally, we turn to pairwise system ranking, developing a method for thresholding performance improvement under an automatic metric against human judgements, which allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected.Together, these findings suggest improvements to the protocols for metric evaluation and system performance evaluation in machine translation.

Accurate, automatic evaluation of machine translation is critical for system tuning, and evaluating progress in the field.We proposed a simple unsupervised metric, and additional supervised metrics which rely on contextual word embeddings to encode the translation and reference sentences.We find that these models rival or surpass all existing metrics in the WMT 2017 sentence-level and systemlevel tracks, and our trained model has a substantially higher correlation with human judgements than all existing metrics on the WMT 2017 to-English sentence level dataset.

Manual data annotation is a vital component of NLP research.When designing annotation tasks, properties of the annotation interface can lead to unintentional artefacts in the resulting dataset, biasing the evaluation.In this paper, we explore sequence effects where annotations of an item are affected by the preceding items.Having assigned one label to an instance, the annotator may be less (or more) likely to assign the same label to the next.During rating tasks, seeing a low quality item may affect the score given to the next item either positively or negatively.We see clear evidence of both types of effects using auto-correlation studies over three different crowdsourced datasets.We then recommend a simple way to minimise sequence effects.

Evaluation of segment-level machine translation metrics is currently hampered by: (1) low inter-annotator agreement levels in human assessments; (2) lack of an effective mechanism for evaluation of translations of equal quality; and (3) lack of methods of significance testing improvements over a baseline.In this paper, we provide solutions to each of these challenges and outline a new human evaluation methodology aimed specifically at assessment of segment-level metrics.We replicate the human evaluation component of WMT-13 and reveal that the current state-of-the-art performance of segment-level metrics is better than previously believed.Three segment-level metrics -METEOR, NLEPOR and SENTBLEU-MOSES -are found to correlate with human assessment at a level not significantly outperformed by any other metric in both the individual language pair assessment for Spanish-to-English and the aggregated set of 9 language pairs.