The meteoric rise in text generation capability has been accompanied by parallel growth in interest in machine-generated text detection: the capability to identify whether a given text was generated using a model or written by a person.While detection models show strong performance, they have the capacity to cause significant negative impacts.We explore potential biases in English machine-generated text detection systems.We curate a dataset of student essays and assess 16 different detection systems for bias across four attributes: gender, race/ethnicity, English-language learner (ELL) status, and economic status.We evaluate these attributes using regression-based models to determine the significance and power of the effects, as well as performing subgroup analysis.We find that while biases are generally inconsistent across systems, there are several key issues: several models tend to classify disadvantaged groups as machine-generated, ELL essays are more likely to be classified as machine-generated, economically disadvantaged students' essays are less likely to be classified as machine-generated, and non-White ELL essays are disproportionately classified as machine-generated relative to their White counterparts.Finally, we perform human annotation and find that while humans perform generally poorly at the detection task, they show no significant biases on the studied attributes.

Jan-Christoph Klie, Ji-Ung Lee, Kevin Stowe, Gözde Şahin, Nafise Sadat Moosavi, Luke Bates, Dominic Petrak, Richard Eckart De Castilho, Iryna Gurevych. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.

Natural language inference (NLI) has been widely used as a task to train and evaluate models for language understanding.However, the ability of NLI models to perform inferences requiring understanding of figurative language such as idioms and metaphors remains understudied.We introduce the IMPLI (Idiomatic and Metaphoric Paired Language Inference) dataset, an English dataset consisting of paired sentences spanning idioms and metaphors.We develop novel methods to generate 24k semiautomatic pairs as well as manually creating 1.8k gold pairs.We use IMPLI to evaluate NLI models based on RoBERTa fine-tuned on the widely used MNLI dataset.We then show that while they can reliably detect entailment relationship between figurative phrases with their literal counterparts, they perform poorly on similarly structured examples where pairs are designed to be non-entailing.This suggests the limits of current NLI models with regard to understanding figurative language and this dataset serves as a benchmark for future improvements in this direction.1

Kevin Stowe, Tuhin Chakrabarty, Nanyun Peng, Smaranda Muresan, Iryna Gurevych. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Metaphor generation is a difficult task, and has seen tremendous improvement with the advent of deep pretrained models.We focus here on the specific task of metaphoric paraphrase generation, in which we provide a literal sentence and generate a metaphoric sentence which paraphrases that input.We compare naive, "free" generation models with those that exploit forms of control over the generation process, adding additional information based on conceptual metaphor theory.We evaluate two methods for generating paired training data, which is then used to train T5 models for free and controlled generation.We use crowdsourcing to evaluate the results, showing that free models tend to generate more fluent paraphrases, while controlled models are better at generating novel metaphors.We then analyze evaluation metrics, showing that different metrics are necessary to capture different aspects of metaphoric paraphrasing.We release our data and models, as well as our annotated results in order to facilitate development of better evaluation metrics. 1

In the field of metaphor detection, deep learning systems are the ubiquitous and achieve strong performance on many tasks.However, due to the complicated procedures for manually identifying metaphors, the datasets available are relatively small and fraught with complications.We show that using syntactic features and lexical resources can automatically provide additional high-quality training data for metaphoric language, and this data can cover gaps and inconsistencies in metaphor annotation, improving state-of-the-art word-level metaphor identification.This novel application of automatically improving training data improves classification across numerous tasks, and reconfirms the necessity of high-quality data for deep learning frameworks.