arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.03350cs.AIcs.CLcs.LG
收藏

miniCTX:基于(长)上下文的神经定理证明

miniCTX: Neural Theorem Proving with (Long-)Contexts

Carnegie Mellon University(卡内基梅隆大学)

Jiewen Hu, Thomas Zhu, Sean Welleck

AI总结:

提出miniCTX基准测试及ntp-toolkit工具,评估神经定理证明器利用长上下文证明训练未见定理的能力,发现基于上下文的方法显著优于传统方法。

摘要:

现实世界中的形式化定理证明通常依赖于大量的上下文信息,包括定义、引理、注释、文件结构以及其他信息。我们引入了miniCTX,用于测试模型证明形式化数学定理的能力,这些定理依赖于训练期间未见过的全新上下文。miniCTX包含源自真实Lean项目和教科书的定理,每个定理都关联着可跨越数万个token的上下文。模型的任务是在能够访问定理所在代码库代码的情况下证明定理,该代码库包含了证明所需的上下文。作为miniCTX的基线,我们测试了微调和提示方法,这些方法以前置上下文为条件进行定理证明。两种方法均显著优于仅依赖状态信息的传统方法。我们发现,miniF2F等先前的基准测试未能捕捉到这种使用上下文的能力。除了miniCTX,我们还提供了ntp-toolkit,用于自动提取和注释定理证明数据,使得将新项目添加到miniCTX中变得容易,以确保训练期间不会看到这些上下文。miniCTX为神经定理证明器提供了具有挑战性且符合现实的评估。

英文摘要:

Real-world formal theorem proving often depends on a wealth of context, including definitions, lemmas, comments, file structure, and other information. We introduce miniCTX, which tests a model's ability to prove formal mathematical theorems that depend on new context that is not seen during training. miniCTX contains theorems sourced from real Lean projects and textbooks, each associated with a context that can span tens of thousands of tokens. Models are tasked with proving a theorem given access to code from the theorem's repository, which contains context that is needed for the proof. As a baseline for miniCTX, we tested fine-tuning and prompting methods that condition theorem proving on preceding context. Both approaches substantially outperform traditional methods that rely solely on state information. We found that this ability to use context is not captured by previous benchmarks such as miniF2F. Alongside miniCTX, we offer ntp-toolkit for automatically extracting and annotating theorem proving data, making it easy to add new projects into miniCTX to ensure that contexts are not seen during training. miniCTX offers a challenging and realistic evaluation of neural theorem provers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑