Codoku：面向前沿编码智能体的可再生程序推理挑战

中文摘要 AI 辅助

现有的程序推理基准要求大型语言模型预测程序在给定输入上的行为。编码智能体打破了这些基准所依赖的两个假设：智能体可以通过执行程序而非推理来恢复答案，并且从现有程序中抽取的固定任务集日益受到污染，且更新成本高昂。我们引入了Codoku（代码数独），这是一个可再生的基准，其中求解器填充部分程序中的类型化单元格，以满足全局静态和动态约束，如规定的控制流图和执行路径。由于部分程序无法执行，且有效填充在指数级大的相互依赖选择空间中稀疏存在，因此工具使用或枚举都无法替代程序推理。谜题通过语义具体化从头合成，因此可以按需生成可控复杂度的新谜题，每个谜题都带有保证可解性的见证。我们通过一个在固定预算内可自由使用任何工具的编码智能体，在300个谜题上评估了五个前沿模型。小型谜题已经对开放权重模型构成挑战，而即使是专有模型也只能解决约一半的大型谜题。因此，Codoku为程序推理提供了一个可再生的测试平台，能够跟上快速改进的编码智能体的步伐。GitHub：此https URL。

英文摘要

Existing program-reasoning benchmarks ask large language models to predict a program's behavior on a given input. Coding agents break two assumptions on which these benchmarks rest: an agent can recover the answer by executing the program instead of reasoning about it, and fixed task sets drawn from existing programs are increasingly exposed to contamination, yet costly to renew. We introduce Codoku (code sudoku), a renewable benchmark in which a solver fills typed cells in a partial program to satisfy global static and dynamic constraints, such as a prescribed control-flow graph and execution path. Because a partial program cannot be executed and valid fillings are sparse in an exponentially large space of interdependent choices, neither tool use nor enumeration can substitute for program reasoning. Puzzles are synthesized from scratch via semantic reification, so fresh puzzles of controllable complexity can be generated on demand, each with a witness that guarantees solvability. We evaluate five frontier models on 300 puzzles through a coding agent free to use any tool within a fixed budget. Small puzzles already challenge open-weight models, whereas even proprietary models solve only about half of the large ones. Codoku thus offers a renewable testbed for program reasoning that can keep pace with rapidly improving coding agents. GitHub: https://github.com/connglli/Codoku.

发表机构

• ETH Zurich（苏黎世联邦理工学院）

机构由 AI 辅助整理，请以论文原文为准。