We investigate a new linguistic generalisation in pre-trained language models (taking BERT Devlin et al. 2019 as a case study).We focus on degree modifiers (expressions like slightly, very, rather, extremely) and test the hypothesis that the degree expressed by a modifier (low, medium or high degree) is related to the modifier's sensitivity to sentence polarity (whether it shows preference for affirmative or negative sentences or neither).To probe this connection, we apply the Artificial Language Learning experimental paradigm from psycholinguistics to a neural language model.Our experimental results suggest that BERT generalizes in line with existing linguistic observations that relate degree semantics to polarity sensitivity, including the main one: low degree semantics is associated with preference towards positive polarity.

Representation of linguistic phenomena in computational language models is typically assessed against the predictions of existing linguistic theories of these phenomena.Using the notion of polarity as a case study, we show that this is not always the most adequate set-up.We probe polarity via so-called 'negative polarity items' (in particular, English any) in two pretrained Transformer-based models (BERT and GPT-2).We show that -at least for polaritymetrics derived from language models are more consistent with data from psycholinguistic experiments than linguistic theory predictions.Establishing this allows us to more adequately evaluate the performance of language models and also to use language models to discover new insights into natural language grammar beyond existing linguistic theories.This work contributes to establishing closer ties between psycholinguistic experiments and experiments with language models.