Multilingual language models often exhibit performance disparities across languages that can arise as early as the tokenization stage.Widelyused subword tokenization approaches favor high-resource languages, and tokenizer-free methods still yield longer sequences for scripts with a higher bytes-per-character ratio.To address these shortcomings, we propose to use the International Phonetic Alphabet (IPA) as a language-agnostic input representation for multilingual tokenizers.IPA provides a compact symbol inventory, greater cross-lingual character overlap, and a more balanced byte-percharacter distribution across languages.We train matched pairs of text vs. IPA subword tokenizers across 24 languages and 14 scripts and demonstrate that IPA tokenizers consistently improve tokenization quality, especially for non-Latin scripts, and generalize more effectively to unseen languages and scripts.Github github.com/Mikki99/ipa-tokenization

The recent advancements in auto-regressive multimodal large language models (MLLMs) have demonstrated promising progress for vision-language tasks. While there exists a variety of studies investigating the processing of linguistic information within large language models, little is currently known about the inner working mechanism of MLLMs and how linguistic and visual information interact within these models. In this study, we aim to fill this gap by examining the information flow between different modalities—language and vision—in MLLMs, focusing on visual question answering. Specifically, given an image-question pair as input, we investigate where in the model and how the visual and linguistic information are combined to generate the final prediction. Conducting experiments with a series of models from the LLaVA series, we find that there are two distinct stages in the process of integration of the two modalities. In the lower layers, the model first transfers the more general visual features of the whole image into the representations of (linguistic) question tokens. In the middle layers, it once again transfers visual information about specific objects relevant to the question to the respective token positions of the question. Finally, in the higher layers, the resulting multimodal representation is propagated to the last position of the input sequence for the final prediction. Overall, our findings provide a new and comprehensive perspective on the spatial and functional aspects of image and language processing in the MLLMs, thereby facilitating future research into multimodal information localization and editing. Our code and collected dataset are released here: https://github.com/FightingFighting/crossmodal-information-flow-in-MLLM.git.

Existing parameter-efficient fine-tuning (PEFT) methods primarily fall into two categories: addition-based and selective in-situ adaptation.The former, such as LoRA, introduce additional modules to adapt the model to downstream tasks, offering strong memory efficiency.However, their representational capacity is often limited, making them less suitable for fine-grained adaptation.In contrast, the latter directly finetunes a carefully chosen subset of the original model parameters, allowing for more precise and effective adaptation, but at the cost of significantly increased memory consumption.To reconcile this trade-off, we propose NeuroAda, a novel PEFT method that enables fine-grained model finetuning while maintaining high memory efficiency.Our approach first identifies important parameters (i.e., connections within the network) as in selective adaptation, and then introduces bypass connections for these selected parameters.During finetuning, only the bypass connections are updated, leaving the original model parameters frozen.Empirical results on 23+ tasks spanning both natural language generation and understanding demonstrate that NeuroAda achieves state-of-the-art performance with as little as ≤ 0.02% trainable parameters, while reducing CUDA memory usage by up to 60%.We release our code here: https://github.com/ FightingFighting/NeuroAda.git.

Texts written in different languages reflect different culturally-dependent beliefs of their writers.Thus, we expect multilingual LMs (MLMs), that are jointly trained on a concatenation of text in multiple languages, to encode different cultural values for each language.Yet, as the 'multilinguality' of these LMs is driven by cross-lingual sharing, we also have reason to belief that cultural values bleed over from one language into another.This limits the use of MLMs in practice, as apart from being proficient in generating text in multiple languages, creating language technology that can serve a community also requires the output of LMs to be sensitive to their biases (Naous et al., 2023).Yet, little is known about how cultural values emerge and evolve in MLMs (Hershcovich et al., 2022a).We are the first to study how languages can exert influence on the cultural values encoded for different test languages, by studying how such values are revised during fine-tuning.Focusing on the finetuning stage allows us to study the interplay between value shifts when exposed to new linguistic experience from different data sources and languages.Lastly, we use a training data attribution method to find patterns in the finetuning examples, and the languages that they come from, that tend to instigate value shifts.

Metaphors in natural language are a reflection of fundamental cognitive processes such as analogical reasoning and categorisation, and are deeply rooted in everyday communication.Metaphor understanding is therefore an essential task for large language models (LLMs).We release the Metaphor Understanding Challenge Dataset (MUNCH), designed to evaluate the metaphor understanding capabilities of LLMs.The dataset provides over 10k paraphrases for sentences containing metaphor use, as well as 1.5k instances containing inapt paraphrases.The inapt paraphrases were carefully selected to serve as control to determine whether the model indeed performs full metaphor interpretation or rather resorts to lexical similarity.All apt and inapt paraphrases were manually annotated.The metaphorical sentences cover natural metaphor uses across 4 genres (academic, news, fiction, and conversation), and they exhibit different levels of novelty.Experiments with LLaMA and GPT-3.5 demonstrate that MUNCH presents a challenging task for LLMs.The dataset is freely accessible at https://github.com/xiaoyuisrain/ metaphor-understanding-challenge.

With the growing size of pre-trained models, full fine-tuning and storing all the parameters for various down-stream tasks is costly and infeasible. In this paper, we propose a new parameter-efficient fine-tuning method, Gradient-based Parameter Selection (GPS), demonstrating that only tuning a few selected parameters from the pre-trained model while keeping the remainder of the model frozen can generate similar or better performance compared with the full model fine-tuning method. Different from the existing popular and state-of-the-art parameter-efficient fine-tuning approaches, our method does not in-troduce any additional parameters and computational costs during both the training and inference stages. Another ad-vantage is the model-agnostic and non-destructive property, which eliminates the need for any other design specific to a particular model. Compared with the full fine-tuning, GPS achieves 3.33% (91.78% vs. 88.45%, FGVC) and 9.61% (73.1% vs. 65.57%, VTAB) improvement of the accu-racy with tuning only 0.36% parameters of the pre-trained model on average over 24 image classification tasks; it also demonstrates a significant improvement of 17% and 16.8% in mDice and mIoU, respectively, on medical image segmentation task. Moreover, GPS achieves state-of-the-art performance compared with existing PEFT meth-ods. The code will be available in https://github.com/FightingFighting/GPS.git.

Simone Tedeschi, Johan Bos, Thierry Declerck, Jan Hajič, Daniel Hershcovich, Eduard Hovy, Alexander Koller, Simon Krek, Steven Schockaert, Rico Sennrich, Ekaterina Shutova, Roberto Navigli. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

We present FewShotTextGCN, a novel method designed to effectively utilize the properties of word-document graphs for improved learning in low-resource settings.We introduce Khop Neighborhood Regularization, a regularizer for heterogeneous graphs, and show that it stabilizes and improves learning when only a few training samples are available.We furthermore propose a simplification in the graphconstruction method, which results in a graph that is ∼7 times less dense and yields better performance in low-resource settings while performing on-par with the state of the art in highresource settings.Finally, we introduce a new variant of Adaptive Pseudo-Labeling tailored for word-document graphs.When using as little as 20 samples for training, we outperform a strong TextGCN baseline with 17% in absolute accuracy on average over eight languages.We demonstrate that our method can be applied to document classification without any language model pretraining on a wide range of typologically diverse languages while performing on par with large pretrained language models.

Multilingual language models (MLMs) are jointly trained on data from many different languages such that representation of individual languages can benefit from other languages' data.Impressive performance in zero-shot cross-lingual transfer shows that these models are able to exploit this property.Yet, it remains unclear to what extent, and under which conditions, languages rely on each other's data.To answer this question, we use TracIn (Pruthi et al., 2020), a training data attribution (TDA) method, to retrieve training samples from multilingual data that are most influential for test predictions in a given language.This allows us to analyse cross-lingual sharing mechanisms of MLMs from a new perspective.While previous work studied cross-lingual sharing at the model parameter level, we present the first approach to study it at the data level.We find that MLMs rely on data from multiple languages during fine-tuning and this reliance increases as finetuning progresses.We further find that training samples from other languages can both reinforce and complement the knowledge acquired from data of the test language itself.

Abstract Large multilingual language models typically share their parameters across all languages, which enables cross-lingual task transfer, but learning can also be hindered when training updates from different languages are in conflict. In this article, we propose novel methods for using language-specific subnetworks, which control cross-lingual parameter sharing, to reduce conflicts and increase positive transfer during fine-tuning. We introduce dynamic subnetworks, which are jointly updated with the model, and we combine our methods with meta-learning, an established, but complementary, technique for improving cross-lingual transfer. Finally, we provide extensive analyses of how each of our methods affects the models.

Anna Langedijk, Verna Dankers, Phillip Lippe, Sander Bos, Bryan Cardenas Guevara, Helen Yannakoudakis, Ekaterina Shutova. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

Abstract Multilingual sentence encoders have seen much success in cross-lingual model transfer for downstream NLP tasks. The success of this transfer is, however, dependent on the model’s ability to encode the patterns of cross-lingual similarity and variation. Yet, we know relatively little about the properties of individual languages or the general patterns of linguistic variation that the models encode. In this article, we investigate these questions by leveraging knowledge from the field of linguistic typology, which studies and documents structural and semantic variation across languages. We propose methods for separating language-specific subspaces within state-of-the-art multilingual sentence encoders (LASER, M-BERT, XLM, and XLM-R) with respect to a range of typological properties pertaining to lexical, morphological, and syntactic structure. Moreover, we investigate how typological information about languages is distributed across all layers of the models. Our results show interesting differences in encoding linguistic variation associated with different pretraining strategies. In addition, we propose a simple method to study how shared typological properties of languages are encoded in two state-of-the-art multilingual models—M-BERT and XLM-R. The results provide insight into their information-sharing mechanisms and suggest that these linguistic properties are encoded jointly across typologically similar languages in these models.

Yingjun Du, Nithin Holla, Xiantong Zhen, Cees Snoek, Ekaterina Shutova. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Rishav Hada, Sohi Sudhir, Pushkar Mishra, Helen Yannakoudakis, Saif M. Mohammad, Ekaterina Shutova. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Computational modelling of political discourse tasks has become an increasingly important area of research in natural language processing.Populist rhetoric has risen across the political sphere in recent years; however, computational approaches to it have been scarce due to its complex nature.In this paper, we present the new Us vs. Them dataset, consisting of 6861 Reddit comments annotated for populist attitudes and the first large-scale computational models of this phenomenon.We investigate the relationship between populist mindsets and social groups, as well as a range of emotions typically associated with these.We set a baseline for two tasks related to populist attitudes and present a set of multi-task learning models that leverage and demonstrate the importance of emotion and group identification as auxiliary tasks.

Niels van der Heijden, Helen Yannakoudakis, Pushkar Mishra, Ekaterina Shutova. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021.

Warning: this paper contains content that may be offensive or upsetting.In this paper, we investigate what types of stereotypical information are captured by pretrained language models.We present the first dataset comprising stereotypical attributes of a range of social groups and propose a method to elicit stereotypes encoded by pretrained language models in an unsupervised fashion.Moreover, we link the emergent stereotypes to their manifestation as basic emotions as a means to study their emotional effects in a more generalized manner.To demonstrate how our methods can be used to analyze emotion and stereotype shifts due to linguistic experience, we use fine-tuning on news sources as a case study.Our experiments expose how attitudes towards different social groups vary across models and how quickly emotions and stereotypes can shift at the fine-tuning stage.

Metaphor is an indispensable part of human cognition and everyday communication.Much research has been conducted elucidating metaphor processing in the mind/brain and the role it plays in communication.In recent years, metaphor processing systems have benefited greatly from these studies, as well as the rapid advances in deep learning for natural language processing (NLP).This paper provides a comprehensive review and discussion of recent developments in automated metaphor processing, in light of the findings about metaphor in the mind, language, and communication, and from the perspective of downstream NLP tasks.

The lack of annotated data in many languages is a well-known challenge within the field of multilingual natural language processing (NLP). Therefore, many recent studies focus on zero-shot transfer learning and joint training across languages to overcome data scarcity for low-resource languages. In this work we (i) perform a comprehensive comparison of state-of-the-art multilingual word and sentence encoders on the tasks of named entity recognition (NER) and part of speech (POS) tagging; and (ii) propose a new method for creating multilingual contextualized word embeddings, compare it to multiple baselines and show that it performs at or above state-of-the-art level in zero-shot transfer settings. Finally, we show that our method allows for better knowledge sharing across languages in a joint training setting.

The rise of online communication platforms has been accompanied by some undesirable effects, such as the proliferation of aggressive and abusive behaviour online.Aiming to tackle this problem, the natural language processing (NLP) community has experimented with a range of techniques for abuse detection.While achieving substantial success, these methods have so far only focused on modelling the linguistic properties of the comments and the online communities of users, disregarding the emotional state of the users and how this might affect their language.The latter is, however, inextricably linked to abusive behaviour.In this paper, we present the first joint model of emotion and abusive language detection, experimenting in a multi-task learning framework that allows one task to inform the other.Our results demonstrate that incorporating affective features leads to significant improvements in abuse detection performance across datasets.

Recent years have seen a growing interest within the natural language processing (NLP) community in evaluating the ability of semantic models to capture human meaning representation in the brain. Existing research has mainly focused on applying semantic models to decode brain activity patterns associated with the meaning of individual words, and, more recently, this approach has been extended to sentences and larger text fragments. Our work is the first to investigate metaphor processing in the brain in this context. We evaluate a range of semantic models (word embeddings, compositional, and visual models) in their ability to decode brain activity associated with reading of both literal and metaphoric sentences. Our results suggest that compositional models and word embeddings are able to capture differences in the processing of literal and metaphoric sentences, providing support for the idea that the literal meaning is not fully accessible during familiar metaphor comprehension.

Recent work shows that distributional semantic models can be used to decode patterns of brain activity associated with individual words and sentence meanings.However, it is yet unclear to what extent such models can be used to study and decode fMRI patterns associated with specific aspects of semantic composition such as the negation function.In this paper, we apply lexical and compositional semantic models to decode fMRI patterns associated with negated and affirmative sentences containing hand-action verbs.Our results show reduced decoding (correlation) of sentences where the verb is in the negated context, as compared to the affirmative one, within brain regions implicated in action-semantic processing.This supports behavioral and brain imaging studies, suggesting that negation involves reduced access to aspects of the affirmative mental representation.The results pave the way for testing alternate semantic models of negation against human semantic processing in the brain.

Verna Dankers, Marek Rei, Martha Lewis, Ekaterina Shutova. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Linguistic typology aims to capture structural and semantic variation across the world’s languages. A large-scale typology could provide excellent guidance for multilingual Natural Language Processing (NLP), particularly for languages that suffer from the lack of human labeled resources. We present an extensive literature survey on the use of typological information in the development of NLP techniques. Our survey demonstrates that to date, the use of information in existing typological databases has resulted in consistent but modest improvements in system performance. We show that this is due to both intrinsic limitations of databases (in terms of coverage and feature granularity) and under-utilization of the typological features included in them. We advocate for a new approach that adapts the broad and discrete nature of typological categories to the contextual and continuous nature of machine learning algorithms used in contemporary NLP. In particular, we suggest that such an approach could be facilitated by recent developments in data-driven induction of typological knowledge.

The rapid growth of social media in recent years has fed into some highly undesirable phenomena such as proliferation of hateful and offensive language on the Internet. Previous research suggests that such abusive content tends to come from users who share a set of common stereotypes and form communities around them. The current state-of-the-art approaches to abuse detection are oblivious to user and community information and rely entirely on textual (i.e., lexical and semantic) cues. In this paper, we propose a novel approach to this problem that incorporates community-based profiling features of Twitter users. Experimenting with a dataset of 16k tweets, we show that our methods significantly outperform the current state of the art in abuse detection. Further, we conduct a qualitative analysis of model characteristics. We release our code, pre-trained models and all the resources used in the public domain.

Metaphors are intriguing.We know that George Lakoff and Mark Johnson, in their book Metaphors We Live By, pointed out that our daily language is full of metaphors (Lakoff and Johnson 1980).Metaphors are not rare at all as linguistic events and in general as a means of human communication (e.g., in visual art).Thus, more than an exception, they seem to be a necessity for our mind.People use metaphors as strategies to link concepts, to deal with situations, and sometimes to suggest solutions to problems.For example, you can see criminality as a monster or as an illness.What you have in mind to deal with it is probably to fight in the former case, and to cure or to plan prevention in the latter.Nonetheless, metaphors are extremely difficult to model in a computational framework.What is literal?What is metaphorical?Making this distinction has often proved to be a daunting task, even for a human judgment.Metaphors-so bound to our way of thinking and entangled with a huge quantity of knowledge and linguistics subtletiesconstitute an excellent research problem for computational linguistics and artificial intelligence in general.We could probably say that they belong to the AI-complete problems.The difficulty of these computational problems is equivalent to that of solving the central artificial intelligence problem-making computers as intelligent as people.Tony Veale, Ekaterina Shutova, and Beata B. Klebanov are experienced researchers in the field of figurative language processing.Their book is an excellent resource and a good reference for anyone who plans to tackle the subtleties of this complex topic.The book offers a comprehensive approach to the computational treatment of metaphors and of related figurative devices such as simile, analogy, and conceptual blending.The reader is introduced to multiple computational perspectives, from symbolic and statistical approaches to interpretation and paraphrase generation, without omitting contributions from philosophy on what constitutes the significance of a metaphor.The first three chapters introduce the reader to the concept of metaphor, particularly the theoretical foundations profitable for approaching the problem computationally.Particularly useful is the explanation of the related figurative devices: similes (the comparison of one thing with another of a different kind, used to make a description more emphatic or vivid, e.g., John is as brave as a lion); analogy (a comparison between one thing and another, typically for the purpose of explanation, e.g., marriage is slavery); and conceptual blending (a cognitive theory, originally developed by Gilles Fauconnier and Mark Turner [Fauconnier and Turner 2002], which refers to a set of cognitive operations for combining-or blending-words, images, and ideas in a network of "mental spaces" to create meaning, e.g., the painting in George Clooney's attic is a cue to create a conceptual

Research in computational semantics is increasingly guided by our understanding of human semantic processing.However, semantic models are typically studied in the context of natural language processing system performance.In this paper, we present a systematic evaluation and comparison of a range of widely-used, stateof-the-art semantic models in their ability to predict patterns of conceptual representation in the human brain.Our results provide new insights both for the design of computational semantic models and for further research in cognitive neuroscience.

The ubiquity of metaphor in our everyday communication makes it an important problem for natural language understanding.Yet, the majority of metaphor processing systems to date rely on handengineered features and there is still no consensus in the field as to which features are optimal for this task.In this paper, we present the first deep learning architecture designed to capture metaphorical composition.Our results demonstrate that it outperforms the existing approaches in the metaphor identification task.

Highly frequent in language and communication, metaphor represents a significant challenge for Natural Language Processing (NLP) applications. Computational work on metaphor has traditionally evolved around the use of hand-coded knowledge, making the systems hard to scale. Recent years have witnessed a rise in statistical approaches to metaphor processing. However, these approaches often require extensive human annotation effort and are predominantly evaluated within a limited domain. In contrast, we experiment with weakly supervised and unsupervised techniques—with little or no annotation—to generalize higher-level mechanisms of metaphor from distributional properties of concepts. We investigate different levels and types of supervision (learning from linguistic examples vs. learning from a given set of metaphorical mappings vs. learning without annotation) in flat and hierarchical, unconstrained and constrained clustering settings. Our aim is to identify the optimal type of supervision for a learning algorithm that discovers patterns of metaphorical association from text. In order to investigate the scalability and adaptability of our models, we applied them to data in three languages from different language groups—English, Spanish, and Russian—achieving state-of-the-art results with little supervision. Finally, we demonstrate that statistical methods can facilitate and scale up cross-linguistic research on metaphor.

Metaphorical expressions are pervasive in natural language and pose a substantial challenge for computational semantics.The inherent compositionality of metaphor makes it an important test case for compositional distributional semantic models (CDSMs).This paper is the first to investigate whether metaphorical composition warrants a distinct treatment in the CDSM framework.We propose a method to learn metaphors as linear transformations in a vector space and find that, across a variety of semantic domains, explicitly modeling metaphor improves the resulting semantic representations.We then use these representations in a metaphor identification task, achieving a high performance of 0.82 in terms of F-score.

Lexico-semantic knowledge of our native language provides an initial foundation for second language learning.In this paper, we investigate whether and to what extent the lexico-semantic models of the native language (L1) are transferred to the second language (L2).Specifically, we focus on the problem of lexical choice and investigate it in the context of three typologically diverse languages: Russian, Spanish and English.We show that a statistical semantic model learned from L1 data improves automatic error detection in L2 for the speakers of the respective L1.Finally, we investigate whether the semantic model learned from a particular L1 is portable to other, typologically related languages.

Metaphor is pervasive in our communication, which makes it an important problem for natural language processing (NLP).Numerous approaches to metaphor processing have thus been proposed, all of which relied on linguistic features and textual data to construct their models.Human metaphor comprehension is, however, known to rely on both our linguistic and perceptual experience, and vision can play a particularly important role when metaphorically projecting imagery across domains.In this paper, we present the first metaphor identification method that simultaneously draws knowledge from linguistic and visual data.Our results demonstrate that it outperforms linguistic and visual models in isolation, as well as being competitive with the best-performing metaphor identification methods, that rely on hand-crafted knowledge about domains and perception.

Understanding cross-cultural differences has important implications for world affairs and many aspects of the life of society. Yet, the majority of text-mining methods to date focus on the analysis of monolingual texts. In contrast, we present a statistical model that simultaneously learns a set of common topics from multilingual, non-parallel data and automatically discovers the differences in perspectives on these topics across linguistic communities. We perform a behavioural evaluation of a subset of the differences identified by our model in English and Spanish to investigate their psychological validity.

Ekaterina Shutova, Niket Tandon, Gerard de Melo. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015.

System design and evaluation methodologies receive significant attention in natural language processing (NLP), with the systems typically being evaluated on a common task and against shared data sets. This enables direct system comparison and facilitates progress in the field. However, computational work on metaphor is considerably more fragmented than similar research efforts in other areas of NLP and semantics. Recent years have seen a growing interest in computational modeling of metaphor, with many new statistical techniques opening routes for improving system accuracy and robustness. However, the lack of a common task definition, shared data set, and evaluation strategy makes the methods hard to compare, and thus hampers our progress as a community in this area. The goal of this article is to review the system features and evaluation strategies that have been proposed for the metaphor processing task, and to analyze their benefits and downsides, with the aim of identifying the desired properties of metaphor processing systems and a set of requirements for their evaluation.

Metaphor is highly frequent in language, which makes its computational processing indispensable for real-world NLP applications addressing semantic tasks. Previous approaches to metaphor modeling rely on task-specific hand-coded knowledge and operate on a limited domain or a subset of phenomena. We present the first integrated open-domain statistical model of metaphor processing in unrestricted text. Our method first identifies metaphorical expressions in running text and then paraphrases them with their literal paraphrases. Such a text-to-text model of metaphor interpretation is compatible with other NLP applications that can benefit from metaphor resolution. Our approach is minimally supervised, relies on the state-of-the-art parsing and lexical acquisition technologies (distributional clustering and selectional preference induction), and operates with a high accuracy.