arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.07646cs.AIcs.CL
收藏

强化学习后训练构建组合推理策略

RL Post-Training Builds Compositional Reasoning Strategies

Azwar Abdulsalam, Nishil Patel, Andrew Saxe

AI总结:

研究强化学习后训练能否组合原始技能成高级策略,通过在可观察环境实验发现,强化学习能解决预训练模型难题,通过分阶段组合机制重组能力,组合过程可复用巩固,与拒绝微调关键差异在选择性,预训练消融表明其对组合策略出现有影响。

摘要:

我们研究强化学习后训练是仅仅放大基础模型中已有的原始技能,还是能将原始技能组合成新的高级策略。在一个完全可观察的重写语法环境中进行研究,预训练分布已知且每次生成的重写都可审核。Transformer在原始符号重写链上进行预训练,在基于跟踪的推理任务上进行后训练,只有二元最终答案奖励。结果表明强化学习解决了预训练模型即使在更大采样预算下仍很少解决的问题,而拒绝微调虽早期有改进但会停滞。跟踪分析显示强化学习通过分阶段组合机制重组原始能力,且组合过程能复用并巩固为稳定的技能库。与拒绝微调相比,关键差异在于选择性而非探索量。预训练消融表明组合策略的出现不仅取决于原始暴露,还取决于预训练是否将原始能力组织成强化学习可压缩的归约过程。基础模型提供了薄弱的程序要素,强化学习将其构建成可靠的高级策略。

英文摘要:

Does RL post-training merely amplify primitive skills already latent in a base model, or can it compose primitive skills into new higher-level strategies? We study this question in a fully observable rewrite-grammar environment where the pretraining distribution is known and every generated rewrite can be audited. A Transformer is pretrained on primitive symbol-rewrite chains and post-trained on a Trace-based reasoning task with only a binary final-answer reward. RL solves held-out problems that remain rarely solved by the pretrained model even under much larger sampling budgets, while rejection fine-tuning improves early but plateaus. Trace analysis shows that RL reorganizes primitive competence through a phased compositional mechanism: it first strengthens primitive reductions, then discovers valid composed procedures. These include sequential compositions, which collapse ordered chains of primitive contractions, and parallel compositions, which combine independent primitive contractions in a single step. The composed procedures are not isolated samples; they are reused and consolidated into a stable repertoire. Comparing RL with rejection fine-tuning shows that the key difference is not exploration volume but selectivity: RFT produces many shortcut-like rewrites, much of them invalid, whereas RL concentrates exploration into valid reusable structure. Pretraining ablations show that the emergence of compositional strategies is gated not by primitive exposure alone, but by whether pretraining organizes primitive competence into reduction procedures that RL can later compress. The base model provides weak procedural ingredients; RL builds them into reliable higher-level strategies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑