发表机构
New York University; Modal Labs; University of California, Los Angeles; University of Illinois Urbana-Champaign; Columbia University（纽约大学; 模态实验室; 加州大学洛杉矶分校; 伊利诺伊大学厄巴纳-香槟分校; 哥伦比亚大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究强化学习在大语言模型从预训练到训练后阶段对推理的作用，以国际象棋为测试平台，按标准流程训练模型，发现预训练损失可预测RL后性能，RL奖励曲线斜率与预训练令牌有关，还揭示RL对SFT策略的影响，且在数学领域也有相同模式。
AI 中文摘要
强化学习（RL）在改进复杂推理任务的大语言模型（LLMs）中变得至关重要，但RL训练后阶段在很大程度上与之前的预训练孤立研究。存在两个基本问题：预训练选择如何影响RL计算回报；RL对模型实际有何作用。在标准LLM设置中难以研究，因此用国际象棋作为可控测试平台。按照标准LLM训练流程，在人类国际象棋游戏上预训练语言模型，在合成推理轨迹上进行监督微调，并在具有可验证奖励的国际象棋谜题上运行RL。结果发现，给定RL计算水平下的RL后性能可由预训练损失很好地预测，RL奖励曲线斜率随预训练令牌近似线性改善。此外，RL不仅锐化了监督微调（SFT）策略，在简单谜题上放大SFT已偏好的正确走法，在难题上揭示SFT下几乎不存在的正确走法。通过在数学领域文本上训练模型进一步测试，发现相同预测模式出现。总之，提供了预训练到RL接口的定量描述和可控测试平台。
英文摘要
Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.
