Large language models are increasingly applied to operational decision-making where the underlying structure is constrained optimization. Existing benchmarks evaluate whether LLMs can formulate optimization problems as solver code, but leave open a complementary question. Can LLMs directly produce correct solutions to fully specified constrained optimization problems without access to a solver? We introduce ConstraintBench, a benchmark for evaluating LLMs on direct constrained optimization across 10 operations research domains, with all ground-truth solutions verified by the Gurobi solver. Each task presents a natural-language scenario with entities, constraints, and an optimization objective; the model must return a structured solution that a deterministic verifier checks against every constraint and the solver-proven optimum. We evaluate six frontier models on 200 tasks and find that feasibility, not optimality, is the primary bottleneck. The best model achieves only 65.0% feasibility, yet feasible solutions average 89 to 96% of the Gurobi-optimal objective. No model exceeds 30.5% on joint feasibility and optimality within 0.1% of the solver reference. Per-domain analysis shows large variation in difficulty, with average feasibility spanning from 85.0% in the facility location domain to 0.8% in the crew assignment domain. Further, systematic failure modes include duration constraint misunderstanding, entity hallucination, and a feasibility-optimality decoupling in facility location and vehicle routing where models achieve high feasibility but 0% optimality. ConstraintBench and all evaluation infrastructure will be publicly released.
Benchmarking Defeasible Reasoning with Large Language Models -- Initial Experiments and Future Directions
School of Computing and Engineering, University of Huddersfield(哈德斯菲尔德大学计算与工程学院);Hellenic Mediterranean University(希腊地中海大学);Leeds Beckett University(利兹贝克特大学)
DeepPHY: Benchmarking Agentic VLMs on Physical Reasoning
1 Taobao \& Tmall Group of Alibaba, 2 Institute of Software, Chinese Academy of Science, 3 University of Chinese Academy of Sciences, 4 Renmin University of China, 5 Informatics Department, PUC-Rio
Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Models
School of Artificial Intelligence, Beihang University, Beijing, China(北京航空航天大学人工智能学院);KoGuan School of Law, Shanghai Jiao Tong University, Shanghai, China(上海交通大学KoGuan法学院);School of Criminal Law, China University of Political Science and Law, Beijing, China(中国政法大学刑事法律学院)
Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimization
Eindhoven University of Technology(埃因霍温理工大学);Brandenburg University of Technology(勃兰登堡工业大学);Nanyang Technological University(南洋理工大学);Delft University of Technology(代尔夫特理工大学);Southeast University(东南大学)
The Measurement Gap in the Automation of EU Law: Benchmarking Doctrinal Legal Reasoning under the EU AI Act
Chair of Law and Artificial Intelligence and Director, CZS Institute for Artificial Intelligence and Law, University of Tübingen(法律与人工智能教授、人工智能与法律研究所主任,图宾根大学)
Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patterns
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Tongyi Lab, Alibaba Group(阿里云实验室)
Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
The Hong Kong Polytechnic University(香港理工大学);University of Electronic Science and Technology of China(电子科技大学);Tencent Youtu Lab(腾讯优图实验室);Beihang University(北京航空航天大学);Simon Fraser University(西蒙弗雷泽大学)
QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation
Huazhong University of Science and Technology(华中科技大学);Nanyang Technological University(南洋理工大学);Hubei University(湖北大学);East China Normal University(华东师范大学);National University of Singapore(新加坡国立大学)
TopoBench: Benchmarking LLMs on Hard Topological Reasoning
Intercom Research(Intercom研究院);University College Dublin(都柏林大学);University of Galway(Galway大学);Salk Institute(Salk研究所);Dublin City University(都柏林城市大学);Trinity College Dublin(都柏林三一学院)
Parameterized Argumentation-based Reasoning Tasks for Benchmarking Generative Language Models
Bernoulli Institute of Mathematics, Computer Science and Artificial Intelligence, University of Groningen(伯努利数学、计算机科学和人工智能研究所,格罗宁根大学);Department of Information and Computing Sciences, Utrecht University(信息与计算科学系,乌得勒支大学)