课程学习可教大语言模型(LLMs)长程隐式推理,但存在上限
Curriculum Learning Teaches LLMs Long-Horizon Implicit Reasoning, Up To A Limit
AI总结:
本研究探究课程学习对 LLMs 长程隐式推理的作用,发现其可提升前瞻步数,且能力可迁移至自然语言推理,但存在固有上限,更大模型拟合上限更高。
摘要:
大语言模型(LLMs)通过思维链(CoT)将中间步骤外化为 token 来进行“推理”,但即使是前沿推理模型在长程任务上也会出现性能骤降。隐式推理(在单次前向传播内完成的推理)在思维链生成过程中十分有用,因为模型可以“向前搜索”以决定正确的下一步,无需显式搜索。隐式推理能力弱的模型更易陷入错误路径,必须通过回溯尝试恢复;而隐式推理能力强的模型可直接识别正确的证明策略,缩短思维链长度。然而,现有研究表明 LLMs 难以在较深的深度下进行隐式推理,且模型规模扩大也无法缓解这一问题。本研究探究课程学习是否可通过在词汇量和前瞻步数(模型为确定正确下一步所需向前搜索的步数)逐步增加的示例上进行微调,来延长该深度。在计算资源匹配的情况下,经课程学习训练的模型可达到无课程学习训练无法实现的前瞻步数。Qwen3-0.6B 经课程学习训练后,在前瞻步数达 256 的问题上准确率达 98%,而无课程学习训练时仅为 45%。不过,随着微调持续推进,进展会放缓,呈现饱和指数趋势,表明模型可学习隐式推理的固有前瞻步数存在上限。更大规模的模型可达到更高的拟合上限,对预训练模型进行微调的效果远优于从头训练,且课程学习步长过大则会阻碍进展。最后,在合成数据上习得的能力可迁移至自然语言推理,在 11 个自然语言推理基准测试中,经课程学习训练的模型在计算资源匹配的情况下表现优于无课程学习训练的模型。
英文摘要:
LLMs "reason" by externalizing intermediate steps as tokens through chain-of-thought (CoT), yet even frontier reasoning models hit abrupt performance cliffs on long-horizon tasks. Implicit reasoning (reasoning within a single forward pass) is useful during CoT generation, as the model can "search ahead" to decide the correct next move without explicit search. Models with weak implicit reasoning are more likely to commit to wrong paths and must attempt to recover via backtracking, while models with strong implicit reasoning identify the correct proof strategy directly, shortening the CoT. However, prior work shows LLMs struggle to reason implicitly beyond shallow depths, with no relief from scale. We investigate whether curriculum learning can extend this depth by fine-tuning on examples with progressively increasing vocabulary size and lookahead (the number of steps a model must search ahead to identify the correct next move). At matched compute, curriculum-trained models achieve lookaheads where no-curriculum training fails. Qwen3-0.6B reaches 98% accuracy on problems with lookahead up to 256 with curriculum, compared to 45% without. However, progress slows as fine-tuning continues, following a saturating exponential that suggests a maximum intrinsic lookahead to which models can learn to reason implicitly. Larger models reach higher fitted ceilings, fine-tuning pretrained models far outpaces training from scratch, and curriculum step sizes that are too large stall progression. Finally, the capability acquired on synthetic data transfers to natural language reasoning, with curriculum-trained models outperforming models trained without curriculum at matched compute on 11 natural language reasoning benchmarks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。