Scaling automatic evaluation of multilingual text generation of LLMs to new tasks, domains, and languages remains a challenge. Traditional evaluation on benchmark datasets carries the risk of reference data leakage in LLM training or involves additional human annotation effort. The alternative strategy of using another LLM as a scorer also faces uncertainty about the ability of this LLM itself to score non-English text. To address these issues, we propose an annotation-free cross-lingual evaluation protocol for multilingual text generation. Given an LLM candidate to be evaluated and a set of non-English inputs for a particular text generation task, our method first generates English references from the translation of the non-English inputs into English. This is done by an LLM that excels in the equivalent English text generation task. The non-English text generated by the LLM candidate is compared against the generated English references using a cross-lingual evaluation metric to assess the ability of the candidate LLM on multilingual text generation. Our protocol shows a high correlation to the reference-based ROUGE metric in four languages on news text summarization. We also evaluate a diverse set of LLMs in over 90 languages with different prompting strategies to study their multilingual generative abilities.

The cascaded approach continues to be the most popular choice for speech translation (ST).This approach consists of an automatic speech recognition (ASR) model and a machine translation (MT) model that are used in a pipeline to translate speech in one language to text in another language.MT models are often trained on well-formed text and therefore lack robustness while translating noisy ASR outputs in the cascaded approach, degrading the overall translation quality significantly.We address this robustness problem in downstream MT models by forcing the MT encoder to bring the representations of a noisy input closer to its clean version in the semantic space.This is achieved by introducing a contrastive learning method that leverages adversarial examples in the form of ASR outputs paired with their corresponding human transcripts to optimize the network parameters.In addition, a curriculum learning strategy is then used to stabilize the training by alternating the standard MT log-likelihood loss and the contrastive losses.Our approach achieves significant gains of up to 3 BLEU scores in English-German and English-French speech translation without hurting the translation quality on clean text.

Large language models (LLMs) like ChatGPT can be expensive to train, deploy, and use for specific natural language generation tasks such as text summarization and for certain domains.A promising alternative is to fine-tune relatively smaller language models (LMs) on a particular task using high-quality, in-domain datasets.However, it can be prohibitively expensive to get such high-quality training data.This issue has been mitigated by generating weakly supervised data via knowledge distillation (KD) of LLMs.We propose a three-step approach to distill ChatGPT and fine-tune smaller LMs for summarizing forum conversations.More specifically, we design a method to selectively sample a large unannotated corpus of forum conversation using a semantic similarity metric.Then, we use the same metric to retrieve suitable prompts for ChatGPT from a small annotated validation set in the same domain.The generated dataset is then filtered to remove lowquality instances.Our proposed select-promptfilter KD approach leads to significant improvements of up to 6.6 ROUGE-2 score by leveraging sufficient in-domain pseudo-labelled data, over a standard KD approach given the same size of training data.

This paper proposes a simple and effective algorithm for incorporating lexical constraints in neural machine translation.Previous work either required re-training existing models with the lexical constraints or incorporating them during beam search decoding with significantly higher computational overheads.Leveraging the flexibility and speed of a recently proposed Levenshtein Transformer model (Gu et al., 2019), our method injects terminology constraints at inference time without any impact on decoding speed.Our method does not require any modification to the training procedure and can be easily applied at runtime with custom dictionaries.Experiments on English-German WMT datasets show that our approach improves an unconstrained baseline and previous approaches.

Automatic post-editing (APE) aims to improve machine translations, thereby reducing human post-editing effort.APE has had notable success when used with statistical machine translation (SMT) systems but has not been as successful over neural machine translation (NMT) systems.This has raised questions on the relevance of APE task in the current scenario.However, the training of APE models has been heavily reliant on large-scale artificial corpora combined with only limited human post-edited data.We hypothesize that APE models have been underperforming in improving NMT translations due to the lack of adequate supervision.To ascertain our hypothesis, we compile a larger corpus of human post-edits of English to German NMT.We empirically show that a state-of-art neural APE model trained on this corpus can significantly improve a strong in-domain NMT system, challenging the current understanding in the field.We further investigate the effects of varying training data sizes, using artificial training data, and domain specificity for the APE task.

Automatic grammatical error correction (GEC) research has made remarkable progress in the past decade.However, all existing approaches to GEC correct errors by considering a single sentence alone and ignoring crucial cross-sentence context.Some errors can only be corrected reliably using cross-sentence context and models can also benefit from the additional contextual information in correcting other errors.In this paper, we address this serious limitation of existing approaches and improve strong neural encoder-decoder models by appropriately modeling wider contexts.We employ an auxiliary encoder that encodes previous sentences and incorporate the encoding in the decoder via attention and gating mechanisms.Our approach results in statistically significant improvements in overall GEC performance over strong baselines across multiple test sets.Analysis of our cross-sentence GEC model on a synthetic dataset shows high performance in verb tense corrections that require cross-sentence context.

We improve automatic correction of grammatical, orthographic, and collocation errors in text using a multilayer convolutional encoder-decoder neural network. The network is initialized with embeddings that make use of character N-gram information to better suit this task. When evaluated on common benchmark test data sets (CoNLL-2014 and JFLEG), our model substantially outperforms all prior neural approaches on this task as well as strong statistical machine translation-based systems with neural and task-specific features trained on the same data. Our analysis shows the superiority of convolutional neural networks over recurrent neural networks such as long short-term memory (LSTM) networks in capturing the local context via attention, and thereby improving the coverage in correcting grammatical errors. By ensembling multiple models, and incorporating an N-gram language model and edit features via rescoring, our novel method becomes the first neural approach to outperform the current state-of-the-art statistical machine translation-based approach, both in terms of grammaticality and fluency.

Several metrics have been proposed for evaluating grammatical error correction (GEC) systems based on grammaticality, fluency, and adequacy of the output sentences. Previous studies of the correlation of these metrics with human quality judgments were inconclusive, due to the lack of appropriate significance tests, discrepancies in the methods, and choice of datasets used. In this paper, we re-evaluate reference-based GEC metrics by measuring the system-level correlations with humans on a large dataset of human judgments of GEC outputs, and by properly conducting statistical significance tests. Our results show no significant advantage of GLEU over MaxMatch (M2), contradicting previous studies that claim GLEU to be superior. For a finer-grained analysis, we additionally evaluate these metrics for their agreement with human judgments at the sentence level. Our sentence-level analysis indicates that comparing GLEU and M2, one metric may be more useful than the other depending on the scenario. We further qualitatively analyze these metrics and our findings show that apart from being less interpretable and non-deterministic, GLEU also produces counter-intuitive scores in commonly occurring test examples.

Grammatical error correction (GEC) systems deployed in language learning environments are expected to accurately correct errors in learners' writing.However, in practice, they often produce spurious corrections and fail to correct many errors, thereby misleading learners.This necessitates the estimation of the quality of output sentences produced by GEC systems so that instructors can selectively intervene and re-correct the sentences which are poorly corrected by the system and ensure that learners get accurate feedback.We propose the first neural approach to automatic quality estimation of GEC output sentences that does not employ any hand-crafted features.Our system is trained in a supervised manner on learner sentences and corresponding GEC system outputs with quality score labels computed using human-annotated references.Our neural quality estimation models for GEC show significant improvements over a strong feature-based baseline.We also show that a state-of-the-art GEC system can be improved when quality scores are used as features for reranking the N-best candidates.

An important aspect for the task of grammatical error correction (GEC) that has not yet been adequately explored is adaptation based on the native language (L1) of writers, despite the marked influences of L1 on second language (L2) writing.In this paper, we adapt a neural network joint model (NNJM) using L1-specific learner text and integrate it into a statistical machine translation (SMT) based GEC system.Specifically, we train an NNJM on general learner text (not L1-specific) and subsequently train on L1-specific data using a Kullback-Leibler divergence regularized objective function in order to preserve generalization of the model.We incorporate this adapted NNJM as a feature in an SMT-based English GEC system and show that adaptation achieves significant F 0.5 score gains on English texts written by L1 Chinese, Russian, and Spanish writers.