Recent work shows strong transfer learning capability to unseen languages in sequence-tosequence neural networks, under the assumption that we have high-quality word representations for the target language.We evaluate whether this direction is a viable path forward for translation from low-resource languages by investigating how much data is required to learn such high-quality word representations.We first show that learning word embeddings separately from a translation model can enable rapid adaptation to new languages with only a few hundred sentences of parallel data.To see whether the current bottleneck in transfer to low-resource languages lies mainly with learning the word representations, we then train word embeddings models on varying amounts of data, to then plug them into a machine translation model.We show that in this simulated low-resource setting with only 500 parallel sentences and 31,250 sentences of monolingual data we can exceed 15 BLEU on Flores on unseen languages.Finally, we investigate why on a real low-resource language the results are less favorable and find fault with the publicly available multilingual language modelling datasets.Lng MADLAD Fineweb CulturaX HPLT clean noisy cs 782,001K hr 48,765K 503,047K kk 46,234K 78,582K 69,353K 51,000K 81,006K is 33,625K 73,760K 48,105K 39,527K 69,643K af 24,339K 64,576K 51,437K 19,537K 37,737K tl 23,639K 97,038K 41,619K 4,516K 52,879K mk 22,537K 48,877K 42,075K 38,494K 57,008K gl 22,214K 78,345K 31,112K 24,524K 61,177K ka 21,008K 56,944K 55,733K 48,299K 63,722K uz 16,581K 28,099K 19,873K 1,152K 14,800K bs 16,561K 217,725K 253,877K 1.2K 268,156K sw 12,839K 27,954K 18,004K 576K 34,308K gu 10,849K 22,527K 20,395K 18,774K 20,639K ur 10,830K 24,907K 43,720K 38,004K 50,629K kn 10,787K 26,427K 24,693K 20,243K 24,929K si 10,777K 20,775K 15,238K 16,332K 33,707K ne 9,535K 18,476K 38,949K 35,581K 37,138K ky 7,402K 14,054K 13,508K 9,295K 10,041K ga 7,155K 124,945K 11,156K 6,108K 10,993K mt 6,442K 18,627K 7,224K 3,337K 8,675K ha 3,560K 7,868K --5,688K ceb 1,677K 10,756K 2,906K 3,375K 2,864K zu 1,320K 8,093K 2,023K -2,710K war 72K 26,042K 2.8K 48K 87k

Multilingual neural machine translation systems learn to map sentences of different languages into a common representation space.Intuitively, with a growing number of seen languages the encoder sentence representation grows more flexible and easily adaptable to new languages.In this work, we test this hypothesis by zero-shot translating from unseen languages.To deal with unknown vocabularies from unknown languages we propose a setup where we decouple learning of vocabulary and syntax, i.e. for each language we learn word representations in a separate step (using cross-lingual word embeddings), and then train to translate while keeping those word representations frozen.We demonstrate that this setup enables zero-shot translation from entirely unseen languages.Zero-shot translating with a model trained on Germanic and Romance languages we achieve scores of 42.6 BLEU for Portuguese-English and 20.7 BLEU for Russian-English on TED domain.We explore how this zero-shot translation capability develops with varying number of languages seen by the encoder.Lastly, we explore the effectiveness of our decoupled learning strategy for unsupervised machine translation.By exploiting our model's zero-shot translation capability for iterative back-translation we attain near parity with a supervised setting.

Tu Anh Dinh, Carlos Mullov, Leonard Bärmann, Zhaolin Li, Danni Liu, Simon Reiß, Jueun Lee, Nathan Lerzer, Jianfeng Gao, Fabian Peller-Konrad, Tobias Röddiger, Alexander Waibel, Tamim Asfour, Michael Beigl, Rainer Stiefelhagen, Carsten Dachsbacher, Klemens Böhm, Jan Niehues. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.