新技能还是更尖锐的原语？从概率角度探讨RLVR中推理能力的出现
New Skills or Sharper Primitives? A Probabilistic Perspective on the Emergence of Reasoning in RLVR
AI总结：
RLVR通过提升原子步骤概率，使模型在多步骤推理中克服指数衰减，展现新能力而非单纯激发潜在痕迹。
摘要：
强化学习与可验证奖励（RLVR）是否赋予大语言模型（LLMs）新的能力，还是仅仅激发潜在痕迹，仍然是一个核心争论。在本工作中，我们支持前者观点，提出一个概率框架，其中能力由实例层面的可解性定义。我们假设复杂推理的出现可以由尖锐化原子步骤的概率驱动，这使模型能够克服多步骤推理链中固有的成功率指数衰减。利用Algebrarium框架，我们仅在单步骤操作上训练模型，并在未见过的多步骤任务上评估其性能。我们的实证结果证实：（1）RLVR通过放大模型现有技能，激励探索之前不可接触的解决方案路径；（2）复合性能严格由原子步骤的联合概率决定，这体现在高皮尔逊相关系数（ρ∈[0.69, 0.96]）上；（3）RLVR作为全局优化器，可以导致特定技能被牺牲以最大化总体奖励。我们的工作为RLVR中涌现能力提供了新的解释，表明可解问题的迭代优化使模型能够开发出解决之前不可解场景的能力。
英文摘要：
Whether Reinforcement Learning with Verifiable Rewards (RLVR) endows Large Language Models (LLMs) with new capabilities or merely elicits latent traces remains a central debate. In this work, we align with the former view, proposing a probabilistic framework where capability is defined by instance-level solvability. We hypothesize that the emergence of complex reasoning can be driven by sharpening atomic step probabilities, which enables models to overcome the exponential decay of success rates inherent in multi-step reasoning chains. Utilizing the Algebrarium framework, we train models exclusively on single-step operations and evaluate their performance on unseen multi-step tasks. Our empirical results confirm that: (1) RLVR incentivizes the exploration of previously inaccessible solution paths by amplifying the model's existing skills; (2) composite performance is strictly governed by the joint probability of atomic steps, evidenced by high Pearson correlation coefficients ($ρ\in [0.69, 0.96]$); and (3) RLVR, acting as a global optimizer, can cause specific skills to be sacrificed to maximize aggregate reward. Our work offers a novel explanation for emergent abilities in RLVR, suggesting that the iterative optimization of solvable problems enables models to develop the capabilities to tackle previously unsolvable scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
