arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.01940cs.CLcs.LGcs.LO
收藏

Lean 4 中的过程驱动自动形式化

Process-Driven Autoformalization in Lean 4

The University of Hong Kong(香港大学);University of Cambridge(剑桥大学);City University of Hong Kong(香港城市大学);Peking University(北京大学);Huawei Hisilicon(华为海思);Sun Yat-sen University(中山大学);Hong Kong University of Science and Technology(香港科技大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))

Jianqiao Lu, Yingjia Wan, Zhengying Liu, Yinya Huang, Jing Xiong, Chengwu Liu, Jianhao Shen, Hui Jin, Jipeng Zhang, Haiming Wang, Zhicheng Yang, Jing Tang, Zhijiang Guo

AI总结:

针对Lean 4自动形式化挑战,提出评估基准与过程监督验证器(PSV)模型,利用编译器反馈有效提升大模型的自动形式化准确率。

摘要:

自动形式化即将自然语言数学转换为形式语言,为推进数学推理提供了巨大潜力。然而,现有的努力仅限于具有大量在线语料库的形式语言,且难以跟上像 Lean 4 这样快速发展的语言。为了弥合这一差距,我们提出了一个新基准 Lean 4 形式化(\ extbf{\ ame}),旨在评估大型语言模型(LLMs)的自动形式化能力。该基准包含对问题、答案、形式陈述和证明的全面评估。此外,我们引入了过程监督验证器(PSV)模型,该模型利用 Lean 4 编译器的精确反馈来增强自动形式化。我们的实验表明,PSV 方法改进了自动形式化,能够使用更少的过滤训练数据实现更高的准确率。此外,当使用包含详细过程信息的数据进行微调时,PSV 可以更有效地利用数据,从而为 Lean 4 带来更显著的自动形式化改进。我们的数据集和代码可在 \url{https://github.com/rookie-joe/PDA} 获取。

英文摘要:

Autoformalization, the conversion of natural language mathematics into formal languages, offers significant potential for advancing mathematical reasoning. However, existing efforts are limited to formal languages with substantial online corpora and struggle to keep pace with rapidly evolving languages like Lean 4. To bridge this gap, we propose a new benchmark \textbf{Form}alization for \textbf{L}ean~\textbf{4} (\textbf{\name}) designed to evaluate the autoformalization capabilities of large language models (LLMs). This benchmark encompasses a comprehensive assessment of questions, answers, formal statements, and proofs. Additionally, we introduce a \textbf{P}rocess-\textbf{S}upervised \textbf{V}erifier (\textbf{PSV}) model that leverages the precise feedback from Lean 4 compilers to enhance autoformalization. Our experiments demonstrate that the PSV method improves autoformalization, enabling higher accuracy using less filtered training data. Furthermore, when fine-tuned with data containing detailed process information, PSV can leverage the data more effectively, leading to more significant improvements in autoformalization for Lean 4. Our dataset and code are available at \url{https://github.com/rookie-joe/PDA}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑