Recent studies have shown that transformer models like BERT rely on number information encoded in their representations of sentences' subjects and head verbs when performing subject-verb agreement.However, probing experiments suggest that subject number is also encoded in the representations of all words in such sentences.In this paper, we use causal interventions to show that BERT only uses the subject plurality information encoded in its representations of the subject and words that agree with it in number.We also demonstrate that current probing metrics are unable to determine which words' representations contain functionally relevant information.This both provides a revised view of subject-verb agreement in language models, and suggests potential pitfalls for current probe usage and evaluation.

The paper explores the ability of LSTM networks trained on a language modeling task to detect linguistic structures which are ungrammatical due to extraction violations (extra arguments and subject-relative clause island violations), and considers its implications for the debate on language innatism. The results show that the current RNN model can correctly classify (un)grammatical sentences, in certain conditions, but it is sensitive to linguistic processing factors and probably ultimately unable to induce a more abstract notion of grammaticality, at least in the domain we tested.

In this study, we use compositional distributional semantic methods to investigate restrictions in adjective ordering.Specifically, we focus on properties distinguishing Adjective-Adjective-Noun phrases in which there is flexibility in the adjective ordering from those bound to a rigid order.We explore a number of measures extracted from the distributional representation of AAN phrases which may indicate a word order restriction.We find that we are able to distinguish the relevant classes and the correct order based primarily on the degree of modification of the adjectives.Our results offer fresh insight into the semantic properties that determine adjective ordering, building a bridge between syntax and distributional semantics.