Multilingual language models (MLLMs) have demonstrated remarkable abilities to transfer knowledge across languages, despite being trained without explicit cross-lingual supervision.We analyze the parameter spaces of three MLLMs to study how their representations evolve during pre-training, observing patterns consistent with compression: models initially form language-specific representations, which gradually converge into cross-lingual abstractions as training progresses.Through probing experiments, we observe a clear transition from uniform language identification capabilities across layers to more specialized layer functions.For deeper analysis, we focus on neurons that encode distinct semantic concepts.By tracing their development during pre-training, we show how they gradually align across languages.Notably, we identify specific neurons that emerge as increasingly reliable predictors for the same concepts across languages.This alignment manifests concretely in generation: once an MLLM exhibits cross-lingual generalization according to our measures, we can select concept-specific neurons identified from, e.g., Spanish text and manipulate them to guide token predictions.Remarkably, rather than generating Spanish text, the model produces semantically coherent English text.This demonstrates that cross-lingually aligned neurons encode generalized semantic representations, independent of the original language encoding.

Recent advances in NLP have led to the creation of powerful language models for many languages including Ancient Greek and Latin.While prior work on Classical languages unanimously uses BERT, in this work we create four language models for Ancient Greek that vary along two dimensions to study their versatility for tasks of interest for Classical languages: we explore (i) encoder-only and encoder-decoder architectures using ROBERTA and T5 as strong model types, and create for each of them (ii) a monolingual Ancient Greek and a multilingual instance that includes Latin and English.We evaluate all models on morphological and syntactic tasks, including lemmatization, which demonstrates the added value of T5's decoding abilities.We further define two probing tasks to investigate the knowledge acquired by models pre-trained on Classical texts.Our experiments provide the first benchmarking analysis of existing models of Ancient Greek.Results show that our models provide significant improvements over the SoTA.The systematic analysis of model types can inform future research in designing language models for Classical languages, including the development of novel generative tasks.We make all our models available as community resources, along with a large curated pre-training corpus for Ancient Greek, to support the creation of a larger, comparable model zoo for Classical Philology.Our models and resources are available at https://github.com/Heidelberg-NLP/ ancient-language-models.