Abstract Despite the success of Transformer-based language models in a wide variety of natural language processing tasks, our understanding of how these models process a given input in order to represent task-relevant information remains incomplete. In this work, we focus on semantic composition and examine how Transformer-based language models represent semantic information related to the meaning of English noun-noun compounds. We probe Transformer-based language models for their knowledge of the thematic relations that link the head nouns and modifier words of compounds (e.g., KITCHEN CHAIR: a chair located in a kitchen). Firstly, using a dataset featuring groups of compounds with shared lexical or semantic features, we find that token representations of six Transformer-based language models distinguish between pairs of compounds based on whether they use the same thematic relation. Secondly, we utilize fine-grained vector representations of compound semantics derived from human annotations, and find that token vectors from several models elicit a strong signal of the semantic relations used in the compounds. In a novel “compositional probe” setting, where we compare the semantic relation signal in mean-pooled token vectors of compounds to mean-pooled token vectors when the two constituent words appear in separate sentences, we find that the Transformer-based language models that best represent the semantics of noun-noun compounds also do so substantially better than in the control condition where the two constituent works are processed separately. Overall, our results shed light on the ability of Transformer-based language models to support compositional semantic processes in representing the meaning of noun-noun compounds.

In this paper we propose a deep learning based approach for image retrieval using EEG. Our approach makes use of a multi-modal deep neural network based on metric learning, where the EEG signal from a user observing an image is mapped together with visual information extracted from the image. The inspiration behind this work is the vision of a system which allows the user to navigate their image catalogue just by thinking about the image they want to see. Thanks to our metric learning approach, the system is scalable in that it can operate with new images that have never been used in training, resulting in a zero-shot image retrieval system. This framework is tested in two different standard EEG image-viewing datasets, where we demonstrate state-of-the-art results in this complex scenario.

Semantic models derived from visual information have helped to overcome some of the limitations of solely text-based distributional semantic models. Researchers have demonstrated that text and image-based representations encode complementary semantic information, which when combined provide a more complete representation of word meaning, in particular when compared with data on human conceptual knowledge. In this work, we reveal that these vision-based representations, whilst quite effective, do not make use of all the semantic information available in the neural network that could be used to inform vector-based models of semantic representation. Instead, we build image-based meta-embeddings from computer vision models, which can incorporate information from all layers of the network, and show that they encode a richer set of semantic attributes and yield a more complete representation of human conceptual knowledge.

Researchers have recently demonstrated that tying the neural weights between the input look-up table and the output classification layer can improve training and lower perplexity on sequence learning tasks such as language modelling.Such a procedure is possible due to the design of the softmax classification layer, which previous work has shown to comprise a viable set of semantic representations for the model vocabulary, and these these output embeddings are known to perform well on word similarity benchmarks.In this paper, we make meaningful comparisons between the input and output embeddings and other SOTA distributional models to gain a better understanding of the types of information they represent.We also construct a new set of word embeddings using the output embeddings to create locally-optimal approximations for the intermediate representations from the language model.These locally-optimal embeddings demonstrate excellent performance across all our evaluations.

Deep learning has become an innovative tool for predicting the properties of a protein. However, obtaining an accurate predictive model using deep learning methods typically requires a large amount of labelled data, which is expensive and time-consuming to accumulate. Even when optimised, these algorithms are often black boxes, which make it challenging to interpret the decision-making processes that lead to the final prediction. Therefore, there is a demand for innovative modelling techniques that overcome these drawbacks within the space of bioinformatic deep learning. To address these issues, we have designed a modelling scheme that utilises techniques from computer vision. Specifically, we explore how triplet-networks can form a robust model architecture that is capable of learning and ranking proteins from just a few labelled examples. We evaluate our model on a variety of downstream tasks, including peak absorption wavelength, enantioselectivity, plasma membrane localisation, and thermostability. The embedded representations produced by this method show considerable improvement when compared to previous baseline models. Finally, to emphasise that this is an example of white-box deep learning, we visualised the features produced by the algorithm to gain a better understanding as to how the network reaches its prediction for each protein property.

Steven Derby, Paul Miller, Barry Devereux. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Distributional models provide a convenient way to model semantics using dense embedding spaces derived from unsupervised learning algorithms.However, the dimensions of dense embedding spaces are not designed to resemble human semantic knowledge.Moreover, embeddings are often built from a single source of information (typically text data), even though neurocognitive research suggests that semantics is deeply linked to both language and perception.In this paper, we combine multimodal information from both text and image-based representations derived from state-of-the-art distributional models to produce sparse, interpretable vectors using Joint Non-Negative Sparse Embedding.Through indepth analyses comparing these sparse models to human-derived behavioural and neuroimaging data, we demonstrate their ability to predict interpretable linguistic descriptions of human ground-truth semantic knowledge.