arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.22465cs.AI
收藏

ConstraintBench: 对直接优化的LLM约束推理进行基准测试

ConstraintBench: Benchmarking LLM Constraint Reasoning on Direct Optimization

Haladir Research Team(Haladir研究团队)

Joseph Tso, Preston Schmittou, Quan Huynh, Jibran Hutchins

AI总结:

ConstraintBench评估LLM在约束优化任务中的直接推理能力,发现可行性是主要瓶颈,最佳模型仅达65%可行性,但可行解接近最优解。

摘要:

大型语言模型越来越多地应用于操作决策中,其中底层结构是约束优化。现有的基准测试评估LLM能否将优化问题转化为求解器代码,但留有补充问题。LLM能否在没有求解器的情况下直接产生完全指定的约束优化问题的正确解?我们介绍了ConstraintBench,这是一个评估LLM在10个运筹学领域直接约束优化的基准,所有真实解均通过Gurobi求解器验证。每个任务提供一个自然语言场景,包含实体、约束和优化目标;模型必须返回一个结构化的解,由确定性验证器检查每个约束和求解器证明的最优解。我们评估了六个前沿模型在200个任务上的表现,发现可行性,而不是最优性,是主要瓶颈。最佳模型仅达到65.0%的可行性,但可行解的平均目标值为Gurobi最优值的89至96%。没有模型在联合可行性与最优性方面超过0.1%的求解器参考值。各领域分析显示难度差异大,可行性平均从设施选址领域85.0%到机组调度领域0.8%。此外,系统性失败模式包括持续约束误解、实体幻觉以及在设施选址和车辆路径中可行性与最优性解耦,其中模型在可行性高但最优性为0%。ConstraintBench和所有评估基础设施将公开发布。

英文摘要:

Large language models are increasingly applied to operational decision-making where the underlying structure is constrained optimization. Existing benchmarks evaluate whether LLMs can formulate optimization problems as solver code, but leave open a complementary question. Can LLMs directly produce correct solutions to fully specified constrained optimization problems without access to a solver? We introduce ConstraintBench, a benchmark for evaluating LLMs on direct constrained optimization across 10 operations research domains, with all ground-truth solutions verified by the Gurobi solver. Each task presents a natural-language scenario with entities, constraints, and an optimization objective; the model must return a structured solution that a deterministic verifier checks against every constraint and the solver-proven optimum. We evaluate six frontier models on 200 tasks and find that feasibility, not optimality, is the primary bottleneck. The best model achieves only 65.0% feasibility, yet feasible solutions average 89 to 96% of the Gurobi-optimal objective. No model exceeds 30.5% on joint feasibility and optimality within 0.1% of the solver reference. Per-domain analysis shows large variation in difficulty, with average feasibility spanning from 85.0% in the facility location domain to 0.8% in the crew assignment domain. Further, systematic failure modes include duration constraint misunderstanding, entity hallucination, and a feasibility-optimality decoupling in facility location and vehicle routing where models achieve high feasibility but 0% optimality. ConstraintBench and all evaluation infrastructure will be publicly released.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑