Existing benchmarks for large language models (LLMs) are largely restricted to high-or mid-resource languages, and often evaluate performance on higher-order tasks in reasoning and generation.However, plenty of evidence points to the fact that LLMs lack basic linguistic competence in the vast majority of the world's 3800+ written languages.We introduce ChiKhaPo, consisting of eight subtasks of varying difficulty designed to evaluate the lexical comprehension and generation abilities of generative models.ChiKhaPo draws on existing lexicons, monolingual data, and bitext, and provides coverage for 2700+ languages for two word-translation-based subtasks, surpassing any existing benchmark in terms of language coverage.We further show that six SOTA models struggle on our benchmark, and discuss the factors contributing to performance scores, including language family, language resourcedness, task, and comprehension versus generation directions.With ChiKhaPo, we hope to enable and encourage the massively multilingual benchmarking of LLMs.

Niyati Bafna, Emily Chang, Nathaniel Romney Robinson, David R. Mortensen, Kenton Murray, David Yarowsky, Hale Sirin. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Marioneta is an installation for the Children's Museum of Pittsburgh which uses the Microsoft Kinect v2 to allow guests to embody a collection of antique puppets in a virtual environment. Final installation in the museum is shown in Fig 1. The focus is on creating an experience wherein elements in the world react to the users' actions through these puppets[1]. Puppet models in the experience are based on a collection of puppets donated to the museum by Margo Lovelace. The original puppets are carefully exhibited in a large display case on the wall in the museum. Many of the puppets in the museum's collection are antiques, fragile or valuable and not suited to hands-on play by the museum's young visitors. Marioneta uses technology to make museum puppets available for imaginative and interesting play[2]. The experience is composed of auto-rotating seasonal stages and season related interactive objects that have visual and audial feedback. Users can throw a pumpkin in fall, pick up an ice ball in winter, play with cowbells in spring, and break a lantern filled with fireflies in summer. One of the stage scenes is shown in Fig 2. Marioneta is an updated version of Virpets, which began in 2001 and remained over 10 years in the museum[3].