Benchmark scores for LLM reasoning systems are reported as single numbers, yet the same model, strategy, and task can produce meaningfully different answers and costs across repeated executions, even under greedy decoding (T = 0). This variance is not a statistical nuisance: the highest-performing strategy wins only 77% of head-to-head runs against its nearest competitor, meaning a single observed score can silently misrank systems. We introduce ReasonBench, a benchmark suite recording 30 independent trials across 10 reasoning strategies, 12 models, and 6 tasks, treating quality and cost as distributions rather than point estimates. We find that this variance is structured rather than random: a two-component taxonomy -- Global Noise, capturing cross-benchmark unevenness, and Run Noise, capturing within-benchmark stochasticity -- reveals that strategy architecture predicts stability profiles, while models and strategies shift orthogonal aspects of the distribution. A hierarchical decomposition attributes three-quarters of score variance to benchmark, system, and item structure, with a persistent residual that single-run evaluation silently absorbs. Finally, cost and quality decouple asymmetrically: cheap methods are structurally immune to joint cost-quality failure, while expensive methods remain exposed regardless of their accuracy. These findings establish instability as an inherent property of reasoning systems and motivate distribution-aware evaluation as standard practice.
Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks
The Measurement Gap in the Automation of EU Law: Benchmarking Doctrinal Legal Reasoning under the EU AI Act
Chair of Law and Artificial Intelligence and Director, CZS Institute for Artificial Intelligence and Law, University of Tübingen(法律与人工智能教授、人工智能与法律研究所主任,图宾根大学)
Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patterns
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Tongyi Lab, Alibaba Group(阿里云实验室)
Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
The Hong Kong Polytechnic University(香港理工大学);University of Electronic Science and Technology of China(电子科技大学);Tencent Youtu Lab(腾讯优图实验室);Beihang University(北京航空航天大学);Simon Fraser University(西蒙弗雷泽大学)
Benchmarking Defeasible Reasoning with Large Language Models -- Initial Experiments and Future Directions
School of Computing and Engineering, University of Huddersfield(哈德斯菲尔德大学计算与工程学院);Hellenic Mediterranean University(希腊地中海大学);Leeds Beckett University(利兹贝克特大学)
TopoBench: Benchmarking LLMs on Hard Topological Reasoning
Intercom Research(Intercom研究院);University College Dublin(都柏林大学);University of Galway(Galway大学);Salk Institute(Salk研究所);Dublin City University(都柏林城市大学);Trinity College Dublin(都柏林三一学院)
Parameterized Argumentation-based Reasoning Tasks for Benchmarking Generative Language Models
Bernoulli Institute of Mathematics, Computer Science and Artificial Intelligence, University of Groningen(伯努利数学、计算机科学和人工智能研究所,格罗宁根大学);Department of Information and Computing Sciences, Utrecht University(信息与计算科学系,乌得勒支大学)
FormalMATH: Benchmarking Formal Mathematical Reasoning of Large Language Models
The Chinese University of Hong Kong(香港中文大学);Numina;Westlake University(西湖大学);M-A-P;University of California, Los Angeles(加州大学洛杉矶分校);Max Planck Institute for Intelligent Systems, Tübingen(蒂宾根马克斯·普朗克智能系统研究所)
DPrivBench: Benchmarking LLMs' Reasoning for Differential Privacy
Halıcıoğlu Data Science Institute, UC San Diego(哈里奇奥格卢数据科学研究所,加州大学圣地亚哥分校);Department of Computer Science and Engineering, UC San Diego(计算机科学与工程系,加州大学圣地亚哥分校);Department of Electrical Engineering, National Taiwan University(电气工程系,国立台湾大学);OpenAI
OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model
Research Center for Social Computing and Interactive Robotics(社会计算与交互机器人研究室);Harbin Institute of Technology(哈尔滨工业大学);Central South University(中南大学);Fudan University(复旦大学);The University of Hong Kong(香港大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Text Computing and Cognitive Intelligence Ministry of Education Engineering Research Center(教育部文本计算与认知智能工程研究中心);Guizhou University(贵州大学)