arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.07372cs.LG
收藏

Transformer树推理后训练中课程的可证明益处

Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Training

cuhk(城市大学);riken(理化学研究所先进智能项目中心);ism(统计数学研究所);usyd(悉尼大学);astar(科技研究局);ntu(南洋理工大学);utokyo(东京大学)

Dake Bu, Wei Huang, Andi Han, Atsushi Nitanda, Hau-San Wong, Qingfu Zhang, Taiji Suzuki

AI总结:

本文提出一个理论框架,证明课程后训练在提升推理性能方面具有指数级改进,通过分析发现课程策略能显著降低样本复杂度。

摘要:

最近在LLM后训练阶段的课程技术已被实验证实能优于非课程方法提升推理性能,但其有效性及局限性仍缺乏系统理解。为弥合这一差距,我们开发了一个抽象理论框架,并确定了课程后训练在样本复杂度上实现指数级改进的充分条件。为了验证该框架,我们将基模型的链式思维生成建模为状态条件化的自回归推理树,并将课程子任务形式化为深度递增课程或提示递减课程。我们的分析显示,使用两种课程策略的强化学习微调能以多项式样本复杂度实现高精度,而非课程对照组则面临指数复杂度瓶颈。我们进一步建立了测试时扩展的类似保证。实验证支持了我们的理论发现。代码可在https://github.com/DakeBU/Curriculum-Post-training获取。

英文摘要:

Recent curriculum techniques in the post-training stage of LLMs have been empirically observed to outperform non-curriculum approaches in improving reasoning performance, yet a principled understanding of their effectiveness and limitations remains incomplete. To bridge this gap, we develop an abstract theoretical framework and identify sufficient conditions under which curriculum post-training yields exponential improvements in sample complexity. To substantiate this framework, we model the base model's Chain-of-Thought generation as a state-conditioned autoregressive reasoning tree, and formalize curriculum subtasks as either depth-increasing curricula that progressively extend reasoning horizons or hint-decreasing curricula that gradually remove partial hints. Our analysis shows that reinforcement learning finetuning with both curriculum strategies achieves high accuracy with polynomial sample complexity, whereas non-curriculum counterpart encounters an exponential complexity bottleneck. We further establish analogous guarantees for test-time scaling. Empirical simulations support our theoretical findings. Code is available at https://github.com/DakeBU/Curriculum-Post-training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑