arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2505.04528cs.AIcs.CLcs.LO
收藏

超越定理证明:形式化问题求解的形式化表述、框架与基准

Beyond Theorem Proving: Formulation, Framework and Benchmark for Formal Problem-Solving

Sch. of Computer Science & Sch. of Artificial Intelligence, Shanghai Jiao Tong University(计算机科学学院与人工智能学院,上海交通大学)

Qi Liu, Xinhao Zheng, Renqiu Xia, Xingzhi Qi, Qinxiang Cao, Junchi Yan

AI总结:

本文将问题求解形式化为确定性马尔可夫决策过程,提出利用形式化定理证明环境进行过程验证的 FPS 与 D-FPS 框架,并构建三个基准及符号化评估方法 RPE,通过基线实验揭示了当前模型在形式化问题求解上的能力差距。

摘要:

作为一个看似不言自明的任务,问题求解一直是科学与工程的重要组成部分。然而,对问题求解本身尚缺乏一个通用而具体的形式化表述。随着基于人工智能的问题求解智能体的近期发展,对过程级可验证性的需求正在迅速增长,但相关探索仍然不足。为填补这些空白,我们提出了将问题求解形式化为确定性马尔可夫决策过程的原则性表述;提出了一个新颖框架 FPS(Formal Problem-Solving,形式化问题求解),该框架利用现有的 FTP(formal theorem proving,形式化定理证明)环境来执行经过过程验证的问题求解;并提出了 D-FPS(Deductive FPS,演绎式形式化问题求解),将求解与答案验证解耦,以实现更好的人类对齐。我们证明了这些框架的表达能力、可靠性和完备性。我们构建了三个问题求解基准:FormalMath500,即 MATH500 基准的一个子集的形式化版本;MiniF2F-Solving 和 PutnamBench-Solving,即对 FTP 基准 MiniF2F 和 PutnamBench 的改编版本。为了实现忠实、可解释且与人类对齐的评估,我们提出了 RPE(Restricted Propositional Equivalence,受限命题等价),这是一种通过形式化验证来确定答案正确性的符号化方法。我们评估了四个主流 FTP 模型和两种提示方法作为基线,它们最多解决了 FormalMath500 的 23.77%、MiniF2F-Solving 的 27.47% 以及 PutnamBench-Solving 的 0.31%。

英文摘要:

As a seemingly self-explanatory task, problem-solving has been a significant component of science and engineering. However, a general yet concrete formulation of problem-solving itself is missing. With the recent development of AI-based problem-solving agents, the demand for process-level verifiability is rapidly increasing yet underexplored. To fill these gaps, we present a principled formulation of problem-solving as a deterministic Markov decision process; a novel framework, FPS (Formal Problem-Solving), which utilizes existing FTP (formal theorem proving) environments to perform process-verified problem-solving; and D-FPS (Deductive FPS), decoupling solving and answer verification for better human-alignment. The expressiveness, soundness and completeness of the frameworks are proven. We construct three benchmarks on problem-solving: FormalMath500, a formalization of a subset of the MATH500 benchmark; MiniF2F-Solving and PutnamBench-Solving, adaptations of FTP benchmarks MiniF2F and PutnamBench. For faithful, interpretable, and human-aligned evaluation, we propose RPE (Restricted Propositional Equivalence), a symbolic approach to determine the correctness of answers by formal verification. We evaluate four prevalent FTP models and two prompting methods as baselines, solving at most 23.77% of FormalMath500, 27.47% of MiniF2F-Solving, and 0.31% of PutnamBench-Solving.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑