Despite the recent success of large language models (LLMs) in reasoning such as DeepSeek, we for the first time identify a key dilemma in reasoning robustness and generalization: significant performance degradation on novel or incomplete data, suggesting a reliance on memorized patterns rather than systematic reasoning. Our closer examination reveals four key unique limitations underlying this issue:(1) Positional bias--models favor earlier queries in multi-query inputs but answering the wrong one in the latter (e.g., GPT-4o's accuracy drops from 75.8 percent to 72.8 percent); (2) Instruction sensitivity--performance declines by 5.0 to 7.5 percent in the Qwen2.5 Series and by 5.0 percent in DeepSeek-V3 with auxiliary guidance; (3) Numerical fragility--value substitution sharply reduces accuracy (e.g., GPT-4o drops from 97.5 percent to 82.5 percent, GPT-o1-mini drops from 97.5 percent to 92.5 percent); and (4) Memory dependence--models resort to guesswork when missing critical data. These findings further highlight the reliance on heuristic recall over rigorous logical inference, demonstrating challenges in reasoning robustness. To comprehensively investigate these robustness challenges, this paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps. This is achieved by an instruction-based approach to generate diverse datasets that closely resemble training distributions, facilitating a holistic robustness assessment and advancing the development of more robust reasoning frameworks. Bad character(s) in field Abstract.
Benchmarking Defeasible Reasoning with Large Language Models -- Initial Experiments and Future Directions
School of Computing and Engineering, University of Huddersfield(哈德斯菲尔德大学计算与工程学院);Hellenic Mediterranean University(希腊地中海大学);Leeds Beckett University(利兹贝克特大学)
DeepPHY: Benchmarking Agentic VLMs on Physical Reasoning
1 Taobao \& Tmall Group of Alibaba, 2 Institute of Software, Chinese Academy of Science, 3 University of Chinese Academy of Sciences, 4 Renmin University of China, 5 Informatics Department, PUC-Rio
Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Models
School of Artificial Intelligence, Beihang University, Beijing, China(北京航空航天大学人工智能学院);KoGuan School of Law, Shanghai Jiao Tong University, Shanghai, China(上海交通大学KoGuan法学院);School of Criminal Law, China University of Political Science and Law, Beijing, China(中国政法大学刑事法律学院)
Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimization
Eindhoven University of Technology(埃因霍温理工大学);Brandenburg University of Technology(勃兰登堡工业大学);Nanyang Technological University(南洋理工大学);Delft University of Technology(代尔夫特理工大学);Southeast University(东南大学)
The Measurement Gap in the Automation of EU Law: Benchmarking Doctrinal Legal Reasoning under the EU AI Act
Chair of Law and Artificial Intelligence and Director, CZS Institute for Artificial Intelligence and Law, University of Tübingen(法律与人工智能教授、人工智能与法律研究所主任,图宾根大学)
Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patterns
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Tongyi Lab, Alibaba Group(阿里云实验室)
Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
The Hong Kong Polytechnic University(香港理工大学);University of Electronic Science and Technology of China(电子科技大学);Tencent Youtu Lab(腾讯优图实验室);Beihang University(北京航空航天大学);Simon Fraser University(西蒙弗雷泽大学)
QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation
Huazhong University of Science and Technology(华中科技大学);Nanyang Technological University(南洋理工大学);Hubei University(湖北大学);East China Normal University(华东师范大学);National University of Singapore(新加坡国立大学)
TopoBench: Benchmarking LLMs on Hard Topological Reasoning
Intercom Research(Intercom研究院);University College Dublin(都柏林大学);University of Galway(Galway大学);Salk Institute(Salk研究所);Dublin City University(都柏林城市大学);Trinity College Dublin(都柏林三一学院)
DPrivBench: Benchmarking LLMs' Reasoning for Differential Privacy
Halıcıoğlu Data Science Institute, UC San Diego(哈里奇奥格卢数据科学研究所,加州大学圣地亚哥分校);Department of Computer Science and Engineering, UC San Diego(计算机科学与工程系,加州大学圣地亚哥分校);Department of Electrical Engineering, National Taiwan University(电气工程系,国立台湾大学);OpenAI
OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model
Research Center for Social Computing and Interactive Robotics(社会计算与交互机器人研究室);Harbin Institute of Technology(哈尔滨工业大学);Central South University(中南大学);Fudan University(复旦大学);The University of Hong Kong(香港大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Text Computing and Cognitive Intelligence Ministry of Education Engineering Research Center(教育部文本计算与认知智能工程研究中心);Guizhou University(贵州大学)