Large language models (LLMs) that fluently converse with humans are a reality -but do LLMs experience human-like processing difficulties?We systematically compare human and LLM sentence comprehension across seven challenging linguistic structures.We collect sentence comprehension data from humans and five families of state-of-the-art LLMs, varying in size and training procedure in a unified experimental framework.Our results show LLMs overall struggle on the target structures, but especially on garden path (GP) sentences.Indeed, while the strongest models achieve near perfect accuracy on non-GP structures (93.7% for GPT-5), they struggle on GP structures (46.8% for GPT-5).Additionally, when ranking structures based on average performance, rank correlation between humans and models increases with parameter count.For each target structure, we also collect data for their matched baseline without the difficult structure.Comparing performance on the target vs. baseline sentences, the performance gap observed in humans holds for LLMs, with two exceptions: for models that are too weak performance is uniformly low across both sentence types, and for models that are too strong the performance is uniformly high.Together, these reveal convergence and divergence in human and LLM sentence comprehension, offering new insights into the similarity of humans and LLMs. 1 Model Average Subj/Obj NP/VP Depth charge NP/S Double center Interference Red.relative GP non GP Human 28.3 13.3 18.5 28.0 29.7 32.3 36.9 41.7 25.8 32.4 o3 74.5 49.0 66.0 64.0 56.0 98.0 100.0 95.0 66.5 87.3 GPT-4.1 68.7 40.0 63.0 63.0 57.0 90.0 100.0 76.0 59.0 84.3 GPT-5 65.6 32.0 45.0 83.0 45.0 98.0 100.0 65.0 46.8 93.7 Llama-11B-Ins.60.6 50.0 72.0 33.0 62.0 61.0 69.0 79.0 65.7 54.3 Llama-11B 59.6 47.0 66.0 38.0 56.0 62.0 82.0 72.0 60.3 60.7 Llama-90B 49.4 25.0 40.0 43.0 29.0 88.0 93.0 39.0 33.3 74.7 DeepSeek-7B 48.0 32.0 41.0 78.0 44.0 25.0 68.0 53.0 42.5 57.0 DeepSeek-1.5B 46.9 38.0 39.0 69.0 39.0 59.0 43.0 40.0 39.0 57.0 DeepSeek-14B 38.6 15.0 25.0 74.0 7.0 66.0 65.0 25.0 18.0 68.3 Qwen-0.6B47.7 36.0 65.0 51.0 39.0 52.0 52.0 40.0 45.0 51.7 Qwen-8B 47.5 28.0 42.0 56.0 33.0 66.0 77.0 38.0 35.3 66.3 Qwen-14B 45.1 12.0 22.0 79.0 10.0 79.0 91.0 34.0 19.5 83.0 Gemma-1B 44.4 40.0 43.0 65.0 38.0 44.0 41.0 39.0 40.0 50.0 Gemma-1B-Ins.39.7 40.0 52.0 64.0 34.0 20.0 28.0 37.0 40.7 37.3 Gemma-4B 38.6 35.0 38.0 55.0 37.0 35.0 35.0 34.0 36.0 41.7

Modern Large Language Models (LLMs) have shown human-like abilities in many language tasks, sparking interest in comparing LLMs’ and humans’ language processing. In this paper, we try to answer two questions: 1. What makes garden-path sentences hard to understand for humans? 2. Do the same reasons make garden-path sentences hard for LLMs as well? Based on psycholinguistic research, we formulate hypotheses on why garden-path sentences are hard, and test these hypotheses on human participants and a large suite of LLMs using comprehension questions. Our findings reveal that both LLMs and humans struggle with specific syntactic complexities, with some models showing high correlation with human comprehension. To complement our findings, we test LLM comprehension of garden-path constructions with paraphrasing and text-to-image generation tasks, and find that the results mirror the sentence comprehension question results, further validating our findings on LLM understanding of these constructions.

Language agents, built on top of language models (LMs), are systems that can interact with complex environments, such as the open web.In this work, we examine whether such agents can perform realistic and time-consuming tasks on the web, e.g., monitoring real-estate markets or locating relevant nearby businesses.We introduce ASSISTANTBENCH, a challenging new benchmark consisting of 214 realistic tasks that can be automatically evaluated, covering different scenarios and domains.We find that AS-SISTANTBENCH exposes the limitations of current systems, including language models and retrieval-augmented language models, as no model reaches an accuracy of more than 25 points.While closed-book LMs perform well in terms of accuracy, they exhibit low precision and tend to hallucinate facts.State-of-the-art web agents reach a score of near zero.Additionally, we introduce SEEPLANACT (SPA), a new web agent that significantly outperforms previous agents, and an ensemble of SPA and closed-book models reaches the best overall performance.Moreover, we analyze failures of current systems and highlight that open web navigation remains a major challenge.

There is increasing interest in using LLMs as decision-making "agents". Doing so includes many degrees of freedom: which model should be used; how should it be prompted; should it be asked to introspect, conduct chain-of-thought reasoning, etc? Settling these questions---and more broadly, determining whether an LLM agent is reliable enough to be trusted---requires a methodology for assessing such an agent's economic rationality. In this paper, we provide one. We begin by surveying the economic literature on rational decision making, taxonomizing a large set of fine-grained "elements" that an agent should exhibit, along with dependencies between them. We then propose a benchmark distribution that quantitatively scores an LLMs performance on these elements and, combined with a user-provided rubric, produces a "rationality report card". Finally, we describe the results of a large-scale empirical experiment with 14 different LLMs, characterizing the both current state of the art and the impact of different model sizes on models' ability to exhibit rational behavior.