arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.02674cs.CLcs.LG
收藏

Mind the Gap:审视大型语言模型的自我改进能力

Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models

Amazon(亚马逊公司);Carnegie Mellon University(卡内基梅隆大学);Harvard University(哈佛大学)

Yuda Song, Hanlin Zhang, Carson Eisenach, Sham Kakade, Dean Foster, Udaya Ghai

AI总结:

本研究提出数学框架,揭示生成-验证差距随预训练算力单调缩放,系统分析LLM自我改进的条件、迭代过程及性能提升方法。

摘要:

自我改进是大型语言模型(LLM)在预训练、后训练和测试时推理中的一种机制。我们探索了一个框架,其中模型验证自己的输出,基于此验证过滤或重新加权数据,并蒸馏过滤后的数据。尽管已有若干经验上的成功,但对其基本原理的理解仍然缺乏。在这项工作中,我们启动了一项关于LLM自我改进的全面、模块化和受控研究。我们为自我改进提供了数学表述,该表述主要受一个我们形式化为“生成-验证差距”的量所支配。通过使用不同模型系列和任务的实验,我们发现了自我改进的一种缩放现象——生成-验证差距的一个变体随模型预训练浮点运算次数单调缩放。我们还考察了自我改进何时可行、迭代自我改进过程以及提升其性能的方法。我们的发现不仅增进了对LLM自我改进的理解并具有实际意义,而且为其能力与边界的未来研究开辟了众多途径。

英文摘要:

Self-improvement is a mechanism in Large Language Model (LLM) pre-training, post-training and test-time inference. We explore a framework where the model verifies its own outputs, filters or reweights data based on this verification, and distills the filtered data. Despite several empirical successes, a fundamental understanding is still lacking. In this work, we initiate a comprehensive, modular and controlled study on LLM self-improvement. We provide a mathematical formulation for self-improvement, which is largely governed by a quantity which we formalize as the generation-verification gap. Through experiments with various model families and tasks, we discover a scaling phenomenon of self-improvement -- a variant of the generation-verification gap scales monotonically with the model pre-training flops. We also examine when self-improvement is possible, an iterative self-improvement procedure, and ways to improve its performance. Our findings not only advance understanding of LLM self-improvement with practical implications, but also open numerous avenues for future research into its capabilities and boundaries.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑