Scaling automatic evaluation of multilingual text generation of LLMs to new tasks, domains, and languages remains a challenge. Traditional evaluation on benchmark datasets carries the risk of reference data leakage in LLM training or involves additional human annotation effort. The alternative strategy of using another LLM as a scorer also faces uncertainty about the ability of this LLM itself to score non-English text. To address these issues, we propose an annotation-free cross-lingual evaluation protocol for multilingual text generation. Given an LLM candidate to be evaluated and a set of non-English inputs for a particular text generation task, our method first generates English references from the translation of the non-English inputs into English. This is done by an LLM that excels in the equivalent English text generation task. The non-English text generated by the LLM candidate is compared against the generated English references using a cross-lingual evaluation metric to assess the ability of the candidate LLM on multilingual text generation. Our protocol shows a high correlation to the reference-based ROUGE metric in four languages on news text summarization. We also evaluate a diverse set of LLMs in over 90 languages with different prompting strategies to study their multilingual generative abilities.

Multilingual neural machine translation (MNMT) offers the convenience of translating between multiple languages with a single model.However, MNMT often suffers from performance degradation in high-resource languages compared to bilingual counterparts.This degradation is commonly attributed to parameter interference, which occurs when parameters are fully shared across all language pairs.In this work, to tackle this issue we propose a gradient-based gradual pruning technique for MNMT.Our approach aims to identify an optimal sub-network for each language pair within the multilingual model by leveraging gradient-based information as pruning criterion and gradually increasing the pruning ratio as schedule.Our approach allows for partial parameter sharing across language pairs to alleviate interference, and each pair preserves its unique parameters to capture language-specific information.Comprehensive experiments on IWSLT and WMT datasets show that our approach yields a notable performance gain on both datasets.

Large language models (LLMs) like ChatGPT can be expensive to train, deploy, and use for specific natural language generation tasks such as text summarization and for certain domains.A promising alternative is to fine-tune relatively smaller language models (LMs) on a particular task using high-quality, in-domain datasets.However, it can be prohibitively expensive to get such high-quality training data.This issue has been mitigated by generating weakly supervised data via knowledge distillation (KD) of LLMs.We propose a three-step approach to distill ChatGPT and fine-tune smaller LMs for summarizing forum conversations.More specifically, we design a method to selectively sample a large unannotated corpus of forum conversation using a semantic similarity metric.Then, we use the same metric to retrieve suitable prompts for ChatGPT from a small annotated validation set in the same domain.The generated dataset is then filtered to remove lowquality instances.Our proposed select-promptfilter KD approach leads to significant improvements of up to 6.6 ROUGE-2 score by leveraging sufficient in-domain pseudo-labelled data, over a standard KD approach given the same size of training data.