Multilingual language models (LMs) organize representations for typologically and orthographically diverse languages into a shared parameter space, yet the nature of this internal organization remains elusive.In this work, we investigate which linguistic properties -abstract language identity or surface-form cues -shape multilingual representations.To do so, we analyze language-associated units across different model families and scales using the Language Activation Probability Entropy (LAPE) metric, and further decompose activations with Sparse Autoencoders.We find that these units are strongly conditioned on orthography: romanization induces near-disjoint representations that align with neither native-script inputs nor English, while word-order shuffling has limited effect on unit identity.Probing shows that typological structure becomes increasingly accessible in deeper layers, while causal interventions indicate that generation is most sensitive to units that are invariant to surface-form perturbations rather than to units identified by typological alignment alone.Overall, our results suggest that multilingual LMs organize representations around surface form, with linguistic abstraction emerging gradually without collapsing into a unified interlingua.

Most multilingual question-answering benchmarks, while covering a diverse pool of languages, do not factor in regional diversity in the information they capture and tend to be Western-centric.This introduces a significant gap in fairly evaluating multilingual models' comprehension of factual information from diverse geographical locations.To address this, we introduce XNationQA for investigating the cultural literacy of multilingual LLMs.XNationQA encompasses a total of 49, 280 questions on the geography, culture, and history of nine countries, presented in seven languages.We benchmark eight standard multilingual LLMs on XNationQA and evaluate them using two novel transference metrics.Our analyses uncover a considerable discrepancy in the models' accessibility to culturally specific facts across languages.Notably, we often find that a model demonstrates greater knowledge of cultural information in English than in the dominant language of the respective culture.The models exhibit better performance in Western languages, although this does not necessarily translate to being more literate for Western countries, which is counterintuitive.Furthermore, we observe that models have a very limited ability to transfer knowledge across languages, particularly evident in open-source models 1 .

Large Language Models (LLMs) have transformed NLP with their remarkable In-context Learning (ICL) capabilities.Automated assistants based on LLMs are gaining popularity; however, adapting them to novel tasks is still challenging.While colossal models excel in zero-shot performance, their computational demands limit widespread use, and smaller language models struggle without context.This paper investigates whether LLMs can generalize from labeled examples of predefined tasks to novel tasks.Drawing inspiration from biological neurons and the mechanistic interpretation of the Transformer architecture, we explore the potential for information sharing across tasks.We design a cross-task prompting setup with three LLMs and show that LLMs achieve significant performance improvements despite no examples from the target task in the context.Cross-task prompting leads to a remarkable performance boost of 107% for LLaMA-2 7B, 18.6% for LLaMA-2 13B, and 3.2% for GPT 3.5 on average over zeroshot prompting, and performs comparable to standard in-context learning.The effectiveness of generating pseudo-labels for in-task examples is demonstrated, and our analyses reveal a strong correlation between the effect of crosstask examples and model activation similarities in source and target input tokens.This paper offers a first-of-its-kind exploration of LLMs' ability to solve novel tasks based on contextual signals from different task examples.Definition: You are given a passage as context and a question related to the passage that can be answered as "True" or "False".Based on the context, question and your reasoning ability answer in a "True" and "False".Context: Many organisms, the action potential is actually initially carried.... Question: Do all neurons have the same action potential?Answer: False Context: Intersex is in some caused by unusual sex hormones.... Question: Can u be born with both male and female parts?Answer: True Context: The Gregorian leap cycle, which has 97 leap days spread.... Question: Are there ever 53 weeks in a year?Answer: True Definition: Given a question from a scientific exam about Physics, Chemistry, and Biology, among others.The question is in multiple choice format with four answer options A., B., C. and D. Using your knowledge about the scientific fields answer the question and provide the label A, B, C and D as answer.Question: Which of the following is not true for myelinated nerve fibers:A. Impulse through ........ neural fibers B. Membrane ...... of Ranvier C. Saltatory .... is seen D. Local anesthesia ....... when the nerve is not covered by myelin sheath Semantically similar example selectionDefinition: You are given a passage as context and a question related to the passage that can be answered as "True" or "False"......