Ivan Vegner, Sydelle De Souza, Valentin Forch, Martha Lewis, Leonidas A. A. Doumas. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Metaphors in natural language are a reflection of fundamental cognitive processes such as analogical reasoning and categorisation, and are deeply rooted in everyday communication.Metaphor understanding is therefore an essential task for large language models (LLMs).We release the Metaphor Understanding Challenge Dataset (MUNCH), designed to evaluate the metaphor understanding capabilities of LLMs.The dataset provides over 10k paraphrases for sentences containing metaphor use, as well as 1.5k instances containing inapt paraphrases.The inapt paraphrases were carefully selected to serve as control to determine whether the model indeed performs full metaphor interpretation or rather resorts to lexical similarity.All apt and inapt paraphrases were manually annotated.The metaphorical sentences cover natural metaphor uses across 4 genres (academic, news, fiction, and conversation), and they exhibit different levels of novelty.Experiments with LLaMA and GPT-3.5 demonstrate that MUNCH presents a challenging task for LLMs.The dataset is freely accessible at https://github.com/xiaoyuisrain/ metaphor-understanding-challenge.

Compositionality enables humans to learn new concepts from their components and the way they combine. Spatial relationships describe how objects are combined in the real world. These are challenging to recognize because they lack a specific shape, yet they communicate information within an image. Recent Vision-Language Models like CLIP [1], a large pre-trained vision and language model, are designed to align images and text. To gain a deeper understanding of CLIP’s capabilities in terms of relationships, we conduct comprehensive experiments to evaluate its performance. We evaluate and fine-tune the CLIP model using a block world dataset. Our findings indicate that CLIP performs poorly when recognizing spatial relationships. Furthermore, various fine-tuned variants can capture some relationships in the training dataset but struggle to generalize to unseen combinations. In an effort to improve CLIP’s performance, we implement several modifications. First, we assemble other models, such as a Symbolic model, which explicitly encodes relationships, as well as pre-trained models like BERT [2] and T5 [3] with the aim of providing additional information and enhancing CLIP’s ability to understand relationship binding. Second, we test two distinct learning strategies, including an auxiliary classifier and a novel compositional contrastive loss. The results show that these methods struggle to generalize learned relationships. Most methods only capture relationships within the training set. While some methods achieve higher scores on the validation and generalization sets, performance on the training set remains relatively low. We also analyze these methods and discuss the types of relationships that the models struggle with, shedding light on potential areas for further improvement.

Metaphor is an indispensable part of human cognition and everyday communication.Much research has been conducted elucidating metaphor processing in the mind/brain and the role it plays in communication.In recent years, metaphor processing systems have benefited greatly from these studies, as well as the rapid advances in deep learning for natural language processing (NLP).This paper provides a comprehensive review and discussion of recent developments in automated metaphor processing, in light of the findings about metaphor in the mind, language, and communication, and from the perspective of downstream NLP tasks.

Words can have multiple senses.Compositional distributional models of meaning have been argued to deal well with finer shades of meaning variation known as polysemy, but are not so well equipped to handle word senses that are etymologically unrelated, or homonymy.Moving from vectors to density matrices allows us to encode a probability distribution over different senses of a word, and can also be accommodated within a compositional distributional model of meaning.In this paper we present three new neural models for learning density matrices from a corpus, and test their ability to discriminate between word senses on a range of compositional datasets.When paired with a particular composition method, our best model outperforms existing vector-based compositional models as well as strong sentence encoders.

Verna Dankers, Marek Rei, Martha Lewis, Ekaterina Shutova. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

We introduce a hierarchical framework for conjunctive concept combination based on conceptual spaces and random set theory. The model has the flexibility to account for composition of concepts at various levels of complexity. We show that the conjunctive model includes linear combination as a special case, and that the more general model can account for non-compositional behaviours such as overextension, non-commutativity, preservation of necessity and impossibility of attributes and to some extent, attribute loss or emergence. We investigate two further aspects of human concept use, the conjunction fallacy and the ‘guppy effect’.