Shaomu Tan, Dawei Zhu, Ke Tran, Michael Denkowski, Sony Trenous, Leonardo F. R. Ribeiro, Bill Byrne, Felix Hieber. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Fine-tuning large language models (LLMs) for machine translation has shown improvements in overall translation quality.However, it is unclear what is the impact of fine-tuning on desirable LLM behaviors that are not present in neural machine translation models, such as steerability, inherent document-level translation abilities, and the ability to produce less literal translations.We perform an extensive translation evaluation on the LLaMA and Falcon family of models with model size ranging from 7 billion up to 65 billion parameters.Our results show that while fine-tuning improves the general translation quality of LLMs, several abilities degrade.In particular, we observe a decline in the ability to perform formality steering, to produce technical translations through few-shot examples, and to perform documentlevel translation.On the other hand, we observe that the model produces less literal translations after fine-tuning on parallel data.We show that by including monolingual data as part of the fine-tuning data we can maintain the abilities while simultaneously enhancing overall translation quality.Our findings emphasize the need for fine-tuning strategies that preserve the benefits of LLMs for machine translation.

Tobias Domhan, Eva Hasler, Ke Tran, Sony Trenous, Bill Byrne, Felix Hieber. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Building neural machine translation systems to perform well on a specific target domain is a well-studied problem.Optimizing system performance for multiple, diverse target domains however remains a challenge.We study this problem in an adaptation setting where the goal is to preserve the existing system quality while incorporating data for domains that were not the focus of the original translation system.We find that we can improve over the performance trade-off offered by Elastic Weight Consolidation with a relatively simple data mixing strategy.At comparable performance on the new domains, catastrophic forgetting is mitigated significantly on strong WMT baselines.Combining both approaches improves the Pareto frontier on this task.