Humans understand sentences word-by-word, in the order that they hear them.This incrementality entails resolving temporary ambiguities about syntactic relationships.We investigate how humans process these syntactic ambiguities by correlating predictions from incremental generative dependency parsers with timecourse data from people undergoing functional neuroimaging while listening to an audiobook.In particular, we compare competing hypotheses regarding the number of developing syntactic analyses in play during word-by-word comprehension: one vs more than one.This comparison involves evaluating syntactic surprisal from a state-of-the-art dependency parser with LLM-adapted encodings against an existing fMRI dataset.In both English and Chinese data, we find evidence for multipath parsing.Brain regions associated with this multipath effect include bilateral superior temporal gyrus.

Do large multilingual language models learn language universals?We consider a much discussed candidate universal, the Final-over-Final Condition (Sheehan et al., 2017b).This Condition is syntactic in the sense that it can only be stated by reference to abstract sentence properties such as nested phrases and head direction.A study of typologically diverse "mixed head direction" languages confirms that the Condition holds in corpora.But in a targeted syntactic evaluation, Gemini Pro only seems to respect the Condition in German, Russian, Hungarian and Serbian.These relatively high-resource languages contrast with Basque, where Gemini Pro does not seem to have learned the Condition at all.This result suggests that modern language models may need additional sources of bias in order to become truly human-like, within a developmentallyrealistic budget of training data.

Neural Language Models (NLMs) have made tremendous advances during the last years, achieving impressive performance on various linguistic tasks. Capitalizing on this, studies in neuroscience have started to use NLMs to study neural activity in the human brain during language processing. However, many questions remain unanswered regarding which factors determine the ability of a neural language model to capture brain activity (aka its ’brain score’). Here, we make first steps in this direction and examine the impact of test loss, training corpus and model architecture (comparing GloVe, LSTM, GPT-2 and BERT), on the prediction of functional Magnetic Resonance Imaging time-courses of participants listening to an audiobook. We find that (1) untrained versions of each model already explain significant amount of signal in the brain by capturing similarity in brain responses across identical words, with the untrained LSTM outperforming the transformer-based models, being less impacted by the effect of context; (2) that training NLP models improves brain scores in the same brain regions irrespective of the model’s architecture; (3) that Perplexity (test loss) is not a good predictor of brain score; (4) that training data have a strong influence on the outcome and, notably, that off-the-shelf models may lack statistical power to detect brain activations. Overall, we outline the impact of model-training choices, and suggest good practices for future studies aiming at explaining the human language system using neural language models.

The Alice Datasets are a set of datasets based on magnetic resonance data and electrophysiological data, collected while participants heard a story in English. Along with the datasets and the text of the story, we provide a variety of different linguistic and computational measures ranging from prosodic predictors to predictors capturing hierarchical syntactic information. These ecologically valid datasets can be easily reused to replicate prior work and to test new hypotheses about natural language comprehension in the brain.

John Hale, Adhiguna Kuncoro, Keith Hall, Chris Dyer, Jonathan Brennan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Recurrent neural network grammars (RNNGs) are generative models of (tree, string) pairs that rely on neural networks to evaluate derivational choices.Parsing with them using beam search yields a variety of incremental complexity metrics such as word surprisal and parser action count.When used as regressors against human electrophysiological responses to naturalistic text, they derive two amplitude effects: an early peak and a P600-like later peak.By contrast, a non-syntactic neural language model yields no reliable effects.Model comparisons attribute the early peak to syntactic composition within the RNNG.This pattern of results recommends the RNNG+beam search combination as a mechanistic model of the syntactic processing that occurs during normal human language comprehension.