Code-switching, alternating between languages within a conversation, is natural for multilingual users, yet poses fundamental challenges for large language models (LLMs).When a user code-switches in their prompt to an LLM, they typically do not specify the expected language of the LLM response, and thus LLMs must infer the output language from contextual and pragmatic cues.We find that current LLMs systematically fail to align with this expectation, responding in undesired languages even when cues are clear to humans.We introduce OLA, a benchmark to evaluate LLMs' Output Language Alignment in code-switched interactions.OLA focuses on Korean-English code-switching and spans simple intra-sentential mixing to instruction-content mismatches.Even frontier models frequently misinterpret implicit language expectations, exhibiting a systematic bias toward non-English responses that generalizes beyond Korean to Chinese and Indonesian pairs.However, Code-Switching Aware DPO with minimal data (∼1K examples) substantially reduces misalignment, suggesting these failures stem from insufficient alignment rather than fundamental limitations.Our results highlight the need to align multilingual LLMs with users' implicit expectations in real-world code-switched interactions.

Large Language Models (LLMs) are increasingly being used to generate synthetic data for training and evaluating models.However, it is unclear whether they can generate a good quality of question answering (QA) dataset that incorporates knowledge and cultural nuance embedded in a language, especially for low-resource languages.In this study, we investigate the effectiveness of using LLMs in generating culturally relevant commonsense QA datasets for Indonesian and Sundanese languages.To do so, we create datasets for these languages using various methods involving both LLMs and human annotators, resulting in ∼4.5K questions per language (∼9K in total), making our dataset the largest of its kind.Our experiments show that automatic data adaptation from an existing English dataset is less effective for Sundanese.Interestingly, using the direct generation method on the target language, GPT-4 Turbo can generate questions with adequate general knowledge in both languages, albeit not as culturally 'deep' as humans.We also observe a higher occurrence of fluency errors in the Sundanese dataset, highlighting the discrepancy between medium-and lower-resource languages.1