arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.06241cs.CL
收藏

重新思考LLM推理中的强化学习:不是能力学习,而是稀疏策略选择

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning

University of Southern California(南加州大学);DEVCOM ARL ReasonMaxxer

Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger, Viktor Prasanna

AI总结:

本文通过分析发现强化学习在LLM推理中主要通过稀疏修正提升性能,提出无需强化学习的ReasonMaxxer方法,仅需少量基础模型推理即可达到与强化学习相当的效果。

摘要:

强化学习已成为提升大语言模型推理能力的标准方法,但证据表明强化学习并未教授新策略,而是重新分配基础模型已含解决方案的概率质量。本文探讨:若强化学习仅引导模型走向其已知路径,强化学习优化循环本身是否必要?通过多模型家族和强化学习算法的token级分析,发现强化学习的有益影响是稀疏、可预测的修正,集中在高熵决策点,模型不确定选择哪条分支。仅1-3%的token位置受扰动,提升的token始终在基础模型的top-5替代选项内,针对这些少数位置的修正可恢复强化学习的大部分准确性,而随机修正无效。基础模型自身的熵可识别这些位置无需强化学习模型,整个修正低维,可表示在极少数模型参数中。这些发现将推理提升重新定义为稀疏策略选择而非能力获取。我们将其转化为ReasonMaxxer,一种无需强化学习的方法,仅在熵门控决策点应用对比损失,使用少量基础模型推理和无在线生成。在三个模型家族、六个规模和六个数学推理基准上,ReasonMaxxer匹配或超过完整强化学习性能,仅需数十问题和分钟单GPU训练,训练成本减少约三个数量级。

英文摘要:

Reinforcement learning has become the standard for improving reasoning in large language models, yet evidence increasingly suggests that RL does not teach new strategies; it redistributes probability mass over solutions the base model already contains. In this work, we ask: if RL merely steers the model toward paths it already knows, is the RL optimization loop itself necessary? Through token-level analysis across multiple model families and RL algorithms, we find that RL's beneficial footprint is a sparse, predictable correction concentrated at high-entropy decision points where the model is uncertain which branch to take. Only 1--3\% of token positions are affected, the promoted token always lies within the base model's top-5 alternatives, and targeted corrections at those few positions causally recover a large fraction of RL's accuracy gain, while random corrections fail. The base model's own entropy identifies these positions without any RL-trained model, and the entire correction is low-dimensional, representable in a tiny fraction of model parameters. These findings reframe reasoning improvement as sparse policy selection, not capability acquisition. We translate this insight into ReasonMaxxer, a minimal RL-free method that applies contrastive loss only at entropy-gated decision points, using a few hundred base-model rollouts and no online generation. Across three model families, six scales, and six math reasoning benchmarks, ReasonMaxxer matches or exceeds full RL performance while requiring only tens of problems and minutes of single-GPU training, a reduction in training cost of roughly three orders of magnitude.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑