Entity typing is central to knowledge engineering, and as large language models (LLMs) are increasingly used to support knowledge graph construction, it becomes essential to assess how reliably they can perform this task. We present HERTy-Wiki (Hierarchical Entity Reasoning and Typing in Wikidata) - a human-verified benchmark for evaluating hierarchical reasoning in LLMs for knowledge graph entity typing (KGET). Unlike existing probing benchmarks that test factual recall, HERTy-Wiki evaluates whether models can select the most specific valid type from limited contextual evidence, reflecting realistic Wikidata scenarios, where editors often work with incomplete or noisy information. HERTy-Wiki comprises 8,767 multiple-choice questions derived from 3,776 Wikidata entities across five domains, with an optional multimodal extension. Each question requires models to discriminate between semantically related types within the same subclass hierarchy. We evaluate several state-of-the-art, reasoning-oriented LLMs under zero-shot, few-shot, and chain-of-thought prompting settings and compare them against a no-context baseline to quantify model reliance on memorised priors. Across models, macro-F1 scores remain around 0.50–0.60, with only modest improvements over the no-context baseline. Our analysis indicates that the emerging hierarchical reasoning abilities of models are overshadowed by strong dependence on memorised priors. These findings highlight fundamental challenges in using LLMs for reasoning in knowledge engineering pipelines, particularly in rapidly evolving or specialised domains with limited pre-training coverage. HERTy-Wiki provides a human-verified benchmark for developing and evaluating genuine reasoning-driven approaches to KGET.

Large language models (LLMs) excel at fewshot learning, but their ability to reject out-ofdistribution examples remains under-explored.We study this challenge under the setting of few-shot open-set classification, where a model must not only classify examples from a small set of seen classes but also reject unseen ones at inference time.This setting is more realistic and challenging than traditional closedset supervised learning, requiring both finegrained classification and robust rejection.We show that, for small LLMs, neither chain-ofthought (CoT) prompting nor supervised finetuning (SFT) alone are sufficient to generalise reliably, particularly when class semantics are anonymised.We introduce Wasserstein GFN (W-GFN), a novel amortised Generative Flow Network framework that uses latent trajectories to approximate the Bayesian posterior.With as few as 4 examples per class, W-GFN substantially improves performance, enabling Llama 3.2 3B to achieve up to ≥ 80% of the performance of Llama 3.3 70B in complex datasets, despite being ∼ 23 times smaller, which highlights the importance of reasoning-aware approaches for robust open-set few-shot learning.

Essay writing is a skill commonly taught and practised in schools. The ability to write a fluent and persuasive essay is often a major component of formal assessment. In natural language processing and education technology we may work with essays in their final form, for example to carry out automated assessment or grammatical error correction. In this work we collect and analyse data representing the essay writing process from start to finish, by recording every key stroke from multiple writers participating in our study. We describe our data collection methodology, the characteristics of the resulting dataset, and the assignment of proficiency levels to the texts. We discuss the ways the keystroke data can be used – for instance seeking to identify patterns in the keystrokes which might act as features in automated assessment or may enable further advancements in writing assistance – and the writing support technology which could be built with such information, if we can detect when writers are struggling to compose a section of their essay and offer appropriate intervention. We frame this work in the context of English language learning, but we note that keystroke logging is relevant more broadly to text authoring scenarios as well as cognitive or linguistic analyses of the writing process.

We present FewShotTextGCN, a novel method designed to effectively utilize the properties of word-document graphs for improved learning in low-resource settings.We introduce Khop Neighborhood Regularization, a regularizer for heterogeneous graphs, and show that it stabilizes and improves learning when only a few training samples are available.We furthermore propose a simplification in the graphconstruction method, which results in a graph that is ∼7 times less dense and yields better performance in low-resource settings while performing on-par with the state of the art in highresource settings.Finally, we introduce a new variant of Adaptive Pseudo-Labeling tailored for word-document graphs.When using as little as 20 samples for training, we outperform a strong TextGCN baseline with 17% in absolute accuracy on average over eight languages.We demonstrate that our method can be applied to document classification without any language model pretraining on a wide range of typologically diverse languages while performing on par with large pretrained language models.

Anna Langedijk, Verna Dankers, Phillip Lippe, Sander Bos, Bryan Cardenas Guevara, Helen Yannakoudakis, Ekaterina Shutova. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

Rishav Hada, Sohi Sudhir, Pushkar Mishra, Helen Yannakoudakis, Saif M. Mohammad, Ekaterina Shutova. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Niels van der Heijden, Helen Yannakoudakis, Pushkar Mishra, Ekaterina Shutova. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021.

We address the task of automatically grading the language proficiency of spontaneous speech based on textual features from automatic speech recognition transcripts.Motivated by recent advances in multi-task learning, we develop neural networks trained in a multi-task fashion that learn to predict the proficiency level of non-native English speakers by taking advantage of inductive transfer between the main task (grading) and auxiliary prediction tasks: morpho-syntactic labeling, language modeling, and native language identification (L1).We encode the transcriptions with both bi-directional recurrent neural networks and with bi-directional representations from transformers, compare against a featurerich baseline, and analyse performance at different proficiency levels and with transcriptions of varying error rates.Our best performance comes from a transformer encoder with L1 prediction as an auxiliary task.We discuss areas for improvement and potential applications for text-only speech scoring.

The rise of online communication platforms has been accompanied by some undesirable effects, such as the proliferation of aggressive and abusive behaviour online.Aiming to tackle this problem, the natural language processing (NLP) community has experimented with a range of techniques for abuse detection.While achieving substantial success, these methods have so far only focused on modelling the linguistic properties of the comments and the online communities of users, disregarding the emotional state of the users and how this might affect their language.The latter is, however, inextricably linked to abusive behaviour.In this paper, we present the first joint model of emotion and abusive language detection, experimenting in a multi-task learning framework that allows one task to inform the other.Our results demonstrate that incorporating affective features leads to significant improvements in abuse detection performance across datasets.

Evaluation of grammatical error correction (GEC) systems has primarily focused on essays written by non-native learners of English, which however is only part of the full spectrum of GEC applications.We aim to broaden the target domain of GEC and release CWEB, a new benchmark for GEC consisting of website text generated by English speakers of varying levels of proficiency.Website data is a common and important domain that contains far fewer grammatical errors than learner essays, which we show presents a challenge to stateof-the-art GEC systems.We demonstrate that a factor behind this is the inability of systems to rely on a strong internal language model in low error density domains.We hope this work shall facilitate the development of opendomain GEC models that generalize to different topics and genres.

We address the task of assessing discourse coherence, an aspect of text quality that is essential for many NLP tasks, such as summarization and language assessment.We propose a hierarchical neural network trained in a multitask fashion that learns to predict a documentlevel coherence score (at the network's top layers) along with word-level grammatical roles (at the bottom layers), taking advantage of inductive transfer between the two tasks.We assess the extent to which our framework generalizes to different domains and prediction tasks, and demonstrate its effectiveness not only on standard binary evaluation coherence tasks, but also on real-world tasks involving the prediction of varying degrees of coherence, achieving a new state of the art.

The rapid growth of social media in recent years has fed into some highly undesirable phenomena such as proliferation of hateful and offensive language on the Internet. Previous research suggests that such abusive content tends to come from users who share a set of common stereotypes and form communities around them. The current state-of-the-art approaches to abuse detection are oblivious to user and community information and rely entirely on textual (i.e., lexical and semantic) cues. In this paper, we propose a novel approach to this problem that incorporates community-based profiling features of Twitter users. Experimenting with a dataset of 16k tweets, we show that our methods significantly outperform the current state of the art in abuse detection. Further, we conduct a qualitative analysis of model characteristics. We release our code, pre-trained models and all the resources used in the public domain.

Youmna Farag, Helen Yannakoudakis, Ted Briscoe. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.

We propose an approach to N -best list reranking using neural sequence-labelling models.We train a compositional model for error detection that calculates the probability of each token in a sentence being correct or incorrect, utilising the full sentence as context.Using the error detection model, we then re-rank the N best hypotheses generated by statistical machine translation systems.Our approach achieves state-of-the-art results on error correction for three different datasets, and it has the additional advantage of only using a small set of easily computed features that require no linguistic input.

Automated Text Scoring (ATS) provides a cost-effective and consistent alternative to human marking.However, in order to achieve good performance, the predictive features of the system need to be manually engineered by human experts.We introduce a model that forms word representations by learning the extent to which specific words contribute to the text's score.Using Long-Short Term Memory networks to represent the meaning of texts, we demonstrate that a fully automated framework is able to achieve excellent results over similar approaches.In an attempt to make our results more interpretable, and inspired by recent advances in visualizing neural networks, we introduce a novel method for identifying the regions of the text that the model has found more discriminative.

In this paper, we present the first experiments using neural network models for the task of error detection in learner writing.We perform a systematic comparison of alternative compositional architectures and propose a framework for error detection based on bidirectional LSTMs.Experiments on the CoNLL-14 shared task dataset show the model is able to outperform other participants on detecting errors in learner writing.Finally, the model is integrated with a publicly deployed self-assessment system, leading to performance comparable to human annotators.