Methods for learning taxonomies from data have been widely studied.We study a specific version of this task, called commonality identification, where only the set of entities is given and we need to find meaningful ways to group those entities.While LLMs should intuitively excel at this task, it is difficult to directly use such models in large domains.In this paper, we instead use LLMs to describe the different properties that are satisfied by each of the entities individually.We then use pretrained embeddings to cluster these properties, and finally group entities that have properties which belong to the same cluster.To achieve good results, it is paramount that the properties predicted by the LLM are sufficiently diverse.We find that this diversity can be improved by prompting the LLM to structure the predicted properties into different facets of knowledge.1

Contextualised Language Models (LM) improve on traditional word embeddings by encoding the meaning of words in context. However, such models have also made it possible to learn high-quality decontextualised concept embeddings. Three main strategies for learning such embeddings have thus far been considered: (i) fine-tuning the LM to directly predict concept embeddings from the name of the concept itself, (ii) averaging contextualised representations of mentions of the concept in a corpus, and (iii) encoding definitions of the concept. As these strategies have complementary strengths and weaknesses, we propose to learn a unified embedding space in which all three types of representations can be integrated. We show that this allows us to outperform existing approaches in tasks such as ontology completion, which heavily depends on access to high-quality concept embeddings. We furthermore find that mentions and definitions are well-aligned in the resulting space, enabling tasks such as target sense verification, even without the need for any fine-tuning.

The theory of Conceptual Spaces is an influential cognitive-linguistic framework for representing the meaning of concepts.Conceptual spaces are constructed from a set of quality dimensions, which essentially correspond to primitive perceptual features (e.g.hue or size).These quality dimensions are usually learned from human judgements, which means that applications of conceptual spaces tend to be limited to narrow domains (e.g.modelling colour or taste).Encouraged by recent findings about the ability of Large Language Models (LLMs) to learn perceptually grounded representations, we explore the potential of such models for learning conceptual spaces.Our experiments show that LLMs can indeed be used for learning meaningful representations to some extent.However, we also find that fine-tuned models of the BERT family are able to match or even outperform the largest GPT-3 model, despite being 2 to 3 orders of magnitude smaller. 1

Concepts play a central role in many applications.This includes settings where concepts have to be modelled in the absence of sentence context.Previous work has therefore focused on distilling decontextualised concept embeddings from language models.But concepts can be modelled from different perspectives, whereas concept embeddings typically mostly capture taxonomic structure.To address this issue, we propose a strategy for identifying what different concepts, from a potentially large concept vocabulary, have in common with others.We then represent concepts in terms of the properties they share with the other concepts.To demonstrate the practical usefulness of this way of modelling concepts, we consider the task of ultra-fine entity typing, which is a challenging multi-label classification problem.We show that by augmenting the label set with shared properties, we can improve the performance of the state-of-the-art models for this task. 1

Grasping the commonsense properties of everyday concepts is an important prerequisite to language understanding. While contextualised language models are reportedly capable of predicting such commonsense properties with human-level accuracy, we argue that such results have been inflated because of the high similarity between training and test concepts. This means that models which capture concept similarity can perform well, even if they do not capture any knowledge of the commonsense properties themselves. In settings where there is no overlap between the properties that are considered during training and testing, we find that the empirical performance of standard language models drops dramatically. To address this, we study the possibility of fine-tuning language models to explicitly model concepts and their properties. In particular, we train separate concept and property encoders on two types of readily available data: extracted hyponym-hypernym pairs and generic sentences. Our experimental results show that the resulting encoders allow us to predict commonsense properties with much higher accuracy than is possible by directly fine-tuning language models. We also present experimental results for the related task of unsupervised hypernym discovery.

Natural Language Inference (NLI) is a fundamental step towards natural language understanding. The task aims to detect whether a premise entails or contradicts a given hypothesis. NLI contributes to a wide range of natural language understanding applications such as question answering, text summarization and information extraction. Recently, the public availability of big datasets such as Stanford Natural Language Inference (SNLI) and SciTail, has made it feasible to train complex neural NLI models. Particularly, Bidirectional Long Short-Term Memory networks (BiLSTMs) with attention mechanisms have shown promising performance for NLI. In this paper, we propose a Combined Attention Model (CAM) for NLI. CAM combines the two attention mechanisms: intra-attention and inter-attention. The model first captures the semantics of the individual input premise and hypothesis with intra-attention and then aligns the premise and hypothesis with inter-sentence attention. We evaluate CAM on two benchmark datasets: Stanford Natural Language Inference (SNLI) and SciTail, achieving 86.14% accuracy on SNLI and 77.23% on SciTail. Further, to investigate the effectiveness of individual attention mechanism and in combination with each other, we present an analysis showing that the intra- and inter-attention mechanisms achieve higher accuracy when they are combined together than when they are independently used.