arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.03203cs.FLcs.AI
收藏

TheoremLlama:将通用大语言模型转化为 Lean4 专家

TheoremLlama: Transforming General-Purpose LLMs into Lean4 Experts

Hong Kong University of Science and Technology(香港科技大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);NVIDIA(英伟达)

Ruida Wang, Jipeng Zhang, Yizhen Jia, Rui Pan, Shizhe Diao, Renjie Pi, Tong Zhang

AI总结:

TheoremLlama 提出端到端框架,通过 NL-FL 数据集自举生成、课程学习与分块训练及迭代证明编写,将通用大语言模型训练为 Lean4 专家,并在 MiniF2F 上超越 GPT-4 基线。

摘要:

使用 Lean 等计算机可验证形式语言证明数学定理对数学推理具有重要影响。形式定理证明的一种方法涉及基于自然语言(NL)证明,利用大语言模型(LLM)生成完整证明。然而,由于对齐的自然语言与形式语言(FL)定理证明数据稀缺,大多数现代大语言模型表现欠佳。这种稀缺性导致训练大语言模型的方法以及充分利用其能力来构造形式证明的技术都很匮乏。为应对这些挑战,本文提出了 TheoremLlama,这是一个端到端框架,用于将通用大语言模型训练为 Lean4 专家。TheoremLlama 包括用于获取对齐数据集的 NL-FL 数据集生成与自举方法、用于训练模型的课程学习与分块训练技术,以及用于编写 Lean4 证明的迭代证明编写方法,这些组件协同工作。利用 TheoremLlama 中的数据集生成方法,我们提供了 Open Bootstrapped Theorems(OBT),一个 NL-FL 对齐且经过自举的数据集。我们新颖的 NL-FL 自举方法将 NL 证明集成到用于训练数据集的 Lean4 代码中,从而利用大语言模型的自然语言推理能力进行形式推理。TheoremLlama 框架在 MiniF2F-Valid 和 Test 数据集上分别取得了 36.48% 和 33.61% 的累积准确率,超过了 GPT-4 基线 22.95% 和 25.41% 的表现。我们的代码、模型检查点以及生成的数据集已在 GitHub 上发布。

英文摘要:

Proving mathematical theorems using computer-verifiable formal languages like Lean significantly impacts mathematical reasoning. One approach to formal theorem proving involves generating complete proofs using Large Language Models (LLMs) based on Natural Language (NL) proofs. However, due to the scarcity of aligned NL and Formal Language (FL) theorem-proving data most modern LLMs exhibit suboptimal performance.This scarcity results in a paucity of methodologies for training LLMs and techniques to fully utilize their capabilities in composing formal proofs. To address these challenges, this paper proposes TheoremLlama, an end-to-end framework that trains a general-purpose LLM to be a Lean4 expert. TheoremLlama includes NL-FL dataset generation and bootstrapping method to obtain aligned dataset, curriculum learning and block training techniques to train the model, and iterative proof writing method to write Lean4 proofs that work together synergistically. Using the dataset generation method in TheoremLlama, we provide Open Bootstrapped Theorems (OBT), an NL-FL aligned and bootstrapped dataset. Our novel NL-FL bootstrapping method, where NL proofs are integrated into Lean4 code for training datasets, leverages the NL reasoning ability of LLMs for formal reasoning. The TheoremLlama framework achieves cumulative accuracies of 36.48% and 33.61% on MiniF2F-Valid and Test datasets respectively, surpassing the GPT-4 baseline of 22.95% and 25.41%. Our code, model checkpoints, and the generated dataset is published in GitHub

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑