arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.20274cs.LGcs.CLcs.SC
收藏

库学习并未实现:关于一次性“库”的奇特案例

Library Learning Doesn't: The Curious Case of the Single-Use "Library"

University of Toronto(多伦多大学);Vector Institute(矢量研究院);Dalhousie University(达尔豪斯大学)

Ian Berlot-Attwell, Frank Rudzicz, Xujie Si

AI总结:

本文考察 LEGO-Prover 与 TroVE 两个数学推理库学习系统,发现其在 miniF2F 和 MATH 上极少复用函数,性能提升主要源于自我纠错与自我一致性。

摘要:

大型语言模型(LLM)的进展掀起了一波用于数学推理的 LLM 库学习系统浪潮。这些系统旨在学习一个可复用的工具库,例如为一系列任务定制的形式化 Isabelle 引理或 Python 程序。许多此类系统的灵感来自人类将知识组织成可复用、可扩展概念的方式,但当前方法是否真的学会了可复用的工具库?我们研究了两个均报告准确率提升的数学库学习系统:LEGO-Prover 和 TroVE。我们发现,在 miniF2F 和 MATH 上,函数复用极其罕见。我们后续的消融实验表明,所观察到的性能提升主要来自自我纠错和自我一致性,而非复用。我们的代码和数据可在 https://github.com/ikb-a/curious-case 获取。

英文摘要:

Advances in Large Language Models (LLMs) have spurred a wave of LLM library learning systems for mathematical reasoning. These systems aim to learn a reusable library of tools, such as formal Isabelle lemmas or Python programs that are tailored to a family of tasks. Many of these systems are inspired by the human structuring of knowledge into reusable and extendable concepts, but do current methods actually learn reusable libraries of tools? We study two library learning systems for mathematics which both reported increased accuracy: LEGO-Prover and TroVE. We find that function reuse is extremely infrequent on miniF2F and MATH. Our followup ablation experiments suggest that, rather than reuse, self-correction and self-consistency are the primary drivers of the observed performance gains. Our code and data are available at https://github.com/ikb-a/curious-case

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑