Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimization
Eindhoven University of Technology（埃因霍温理工大学）；Brandenburg University of Technology（勃兰登堡工业大学）；Nanyang Technological University（南洋理工大学）；Delft University of Technology（代尔夫特理工大学）；Southeast University（东南大学）
While large language models (LLMs) have shown strong performance in math and logic reasoning, their ability to handle combinatorial optimization (CO) -- searching high-dimensional solution spaces under hard constraints -- remains underexplored. To bridge the gap, we introduce NLCO, a \textbf{N}atural \textbf{L}anguage \textbf{C}ombinatorial \textbf{O}ptimization benchmark that evaluates LLMs on end-to-end CO reasoning: given a language-described decision-making scenario, the model must output a discrete solution without writing code or calling external solvers. NLCO covers 43 CO problems and is organized using a four-layer taxonomy of variable types, constraint families, global patterns, and objective classes, enabling fine-grained evaluation. We provide solver-annotated solutions and comprehensively evaluate LLMs by feasibility, solution optimality, and reasoning efficiency. Experiments across a wide range of modern LLMs show that high-performing models achieve strong feasibility and solution quality on small instances, but both degrade as instance size grows, even if more tokens are used for reasoning. We also observe systematic effects across the taxonomy: set-based tasks are relatively easy, whereas graph-structured problems and bottleneck objectives lead to more frequent failures.
Benchmarking Defeasible Reasoning with Large Language Models -- Initial Experiments and Future Directions
School of Computing and Engineering, University of Huddersfield（哈德斯菲尔德大学计算与工程学院）；Hellenic Mediterranean University（希腊地中海大学）；Leeds Beckett University（利兹贝克特大学）
DeepPHY: Benchmarking Agentic VLMs on Physical Reasoning
1 Taobao \& Tmall Group of Alibaba, 2 Institute of Software, Chinese Academy of Science, 3 University of Chinese Academy of Sciences, 4 Renmin University of China, 5 Informatics Department, PUC-Rio
Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Models
School of Artificial Intelligence, Beihang University, Beijing, China（北京航空航天大学人工智能学院）；KoGuan School of Law, Shanghai Jiao Tong University, Shanghai, China（上海交通大学KoGuan法学院）；School of Criminal Law, China University of Political Science and Law, Beijing, China（中国政法大学刑事法律学院）
The Measurement Gap in the Automation of EU Law: Benchmarking Doctrinal Legal Reasoning under the EU AI Act
Chair of Law and Artificial Intelligence and Director, CZS Institute for Artificial Intelligence and Law, University of Tübingen（法律与人工智能教授、人工智能与法律研究所主任，图宾根大学）
Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patterns
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Tongyi Lab, Alibaba Group（阿里云实验室）
Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
The Hong Kong Polytechnic University（香港理工大学）；University of Electronic Science and Technology of China（电子科技大学）；Tencent Youtu Lab（腾讯优图实验室）；Beihang University（北京航空航天大学）；Simon Fraser University（西蒙弗雷泽大学）
QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation
Huazhong University of Science and Technology（华中科技大学）；Nanyang Technological University（南洋理工大学）；Hubei University（湖北大学）；East China Normal University（华东师范大学）；National University of Singapore（新加坡国立大学）
TopoBench: Benchmarking LLMs on Hard Topological Reasoning
Intercom Research（Intercom研究院）；University College Dublin（都柏林大学）；University of Galway（Galway大学）；Salk Institute（Salk研究所）；Dublin City University（都柏林城市大学）；Trinity College Dublin（都柏林三一学院）
