arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.22117cs.LGcs.AI
收藏

关于RLVR更新方向的LLM推理:识别与利用

On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation

Alibaba Group(阿里巴巴集团)

Kexin Huang, Haoming Meng, Junkang Wu, Jinda Lu, Chiyu Ma, Ziqian Chen, Xue Wang, Bolin Ding, Jiancan Wu, Xiang Wang, Xiangnan He, Guoyin Wang, Jingren Zhou

AI总结:

本文探讨RLVR更新方向对LLM推理的影响,提出通过Δlogp识别关键更新,并应用于测试时插值和训练时重加权以提升推理性能。

摘要:

可验证奖励强化学习(RLVR)显著提升了大语言模型的推理能力。尽管现有分析指出RLVR引起的更改稀疏,但主要关注更新的幅度,忽视了方向。本文认为更新方向是理解RLVR效果的关键,可通过基模型与最终RLVR模型之间的符号化、令牌级对数概率差异Δlogp捕捉。通过统计分析和令牌替换干预,证明Δlogp比基于幅度的指标更有效识别稀疏但推理关键的更新。基于此,提出两种应用:(1)测试时插值方法,沿学习的Δlogp方向增强策略以提高推理准确性;(2)训练时重加权方法,聚焦低概率(对应高Δlogp)令牌,提升模型和基准的推理性能。本文确立了变化方向作为分析和改进RLVR的关键原则。

英文摘要:

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models. While existing analyses identify that RLVR-induced changes are sparse, they primarily focus on the \textbf{magnitude} of these updates, largely overlooking their \textbf{direction}. In this work, we argue that the direction of updates is a more critical lens for understanding RLVR's effects, which can be captured by the signed, token-level log probability difference $Δ\log p$ between the base and final RLVR models. Through statistical analysis and token-replacement interventions, we demonstrate that $Δ\log p$ more effectively identifies sparse, yet reasoning-critical updates than magnitude-based metrics (\eg divergence or entropy). Building on this insight, we propose two practical applications: (1) a \textit{test-time extrapolation} method that amplifies the policy along the learned $Δ\log p$ direction to improve reasoning accuracy without further training; (2) a \textit{training-time reweighting} method that focuses learning on low-probability (corresponding to higher $Δ\log p$) tokens, which improves reasoning performance across models and benchmarks. Our work establishes the direction of change as a key principle for analyzing and improving RLVR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑