Large language models demonstrate remarkable capabilities across various domains, especially mathematics and logic reasoning. However, current evaluations overlook physics-based reasoning - a complex task requiring physics theorems and constraints. We present PhysReason, a 1,200-problem benchmark comprising knowledge-based (25%) and reasoning-based (75%) problems, where the latter are divided into three difficulty levels (easy, medium, hard). Notably, problems require an average of 8.1 solution steps, with hard requiring 15.6, reflecting the complexity of physics-based reasoning. We propose the Physics Solution Auto Scoring Framework, incorporating efficient answer-level and comprehensive step-level evaluations. Top-performing models like Deepseek-R1, Gemini-2.0-Flash-Thinking, and o3-mini-high achieve less than 60% on answer-level evaluation, with performance dropping from knowledge questions (75.11%) to hard problems (31.95%). Through step-level evaluation, we identified four key bottlenecks: Physics Theorem Application, Physics Process Understanding, Calculation, and Physics Condition Analysis. These findings position PhysReason as a novel and comprehensive benchmark for evaluating physics-based reasoning capabilities in large language models.

Xinyu Zhang, Lingling Zhang, Yanrui Wu, Muye Huang, Wenjun Wu, Bo Li, Shaowei Wang, Basura Fernando, Jun Liu. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Abstract visual reasoning, exemplified by Raven's Progressive Matrices (RPM), remains a significant challenge in artificial intelligence. A critical difficulty lies in disentangling abstract relational rules from image-specific features, as these rules operate independently of visual appearances. To address this challenge, we propose the Cognitive Predictive Coding Network (CPCN), inspired by predictive coding theory from cognitive science. CPCN features a three-component architecture: a Relation Disentangler that separates abstract rules from image-specific features through prediction error minimization; Stacked Free Energy Minimizers (FEMs) that leverage energy minimization principles to progressively reduce uncertainty during hierarchical abstraction; and a classifier for solution identification. Unlike previous approaches, our model employs mutual information constraints to explicitly separate relation-relevant and relation-irrelevant features, enabling more robust pattern recognition. Our novel FEMs provide a principled approach to uncertainty reduction through iterative refinement of pattern understanding. Experiments demonstrate PAH's superior performance across multiple benchmarks (such as 98.9% on RAVEN-Fair) with state-of-the-art 59.7% average accuracy across all PGM subtasks.