arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.10014cs.LGstat.ML
收藏

一种以任务为中心的迭代自改进理论,采用由易到难的课程

A Task-Centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula

New York University(纽约大学)

Chenruo Liu, Yijun Dong, Yiqiu Shen, Qi Lei

AI总结:

本文通过将自改进建模为基于奖励过滤分布的最大似然微调,推导了期望奖励的有限样本保证,并证明了在推理任务中由易到难的课程比固定任务混合训练具有更好的理论保证。

摘要:

迭代自改进在由大型语言模型自身生成的、经过奖励验证的输出上微调自回归大型语言模型。与自改进的经验成功相比,这种生成性迭代过程在实际有限样本设置下的理论基础仍然有限。我们通过将每一轮自改进建模为在奖励过滤分布上的最大似然微调,并推导期望奖励的有限样本保证,朝这个目标取得了进展。我们的分析揭示了一个显式的反馈循环,其中更好的模型每轮接受更多数据,支持持续的自改进,同时解释了这种改进最终饱和的原因。通过采用以任务为中心的观点,考虑具有多个难度级别的推理任务,我们进一步证明了在模型初始化、任务难度和样本预算方面的可量化条件,在这些条件下,由易到难的课程比在固定任务混合上训练具有可证明的更好保证。我们的分析通过蒙特卡洛模拟以及涵盖合成图基推理任务和多个标准数学推理基准的实验得到了验证。

英文摘要:

Iterative self-improvement fine-tunes an autoregressive large language model (LLM) on reward-verified outputs generated by the LLM itself. In contrast to the empirical success of self-improvement, the theoretical foundation of this generative, iterative procedure in a practical, finite-sample setting remains limited. We make progress toward this goal by modeling each round of self-improvement as maximum-likelihood fine-tuning on a reward-filtered distribution and deriving finite-sample guarantees for the expected reward. Our analysis reveals an explicit feedback loop where better models accept more data per iteration, supporting sustained self-improvement while explaining eventual saturation of such improvement. Adopting a task-centric view by considering reasoning tasks with multiple difficulty levels, we further prove quantifiable conditions on model initialization, task difficulty, and sample budget where easy-to-hard curricula provably achieve better guarantees than training on fixed mixtures of tasks. Our analyses are validated through Monte-Carlo simulations and experiments spanning a synthetic graph-based reasoning task and multiple standard mathematical reasoning benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑