Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models. While existing analyses identify that RLVR-induced changes are sparse, they primarily focus on the \textbf{magnitude} of these updates, largely overlooking their \textbf{direction}. In this work, we argue that the direction of updates is a more critical lens for understanding RLVR's effects, which can be captured by the signed, token-level log probability difference $Δ\log p$ between the base and final RLVR models. Through statistical analysis and token-replacement interventions, we demonstrate that $Δ\log p$ more effectively identifies sparse, yet reasoning-critical updates than magnitude-based metrics (\eg divergence or entropy). Building on this insight, we propose two practical applications: (1) a \textit{test-time extrapolation} method that amplifies the policy along the learned $Δ\log p$ direction to improve reasoning accuracy without further training; (2) a \textit{training-time reweighting} method that focuses learning on low-probability (corresponding to higher $Δ\log p$) tokens, which improves reasoning performance across models and benchmarks. Our work establishes the direction of change as a key principle for analyzing and improving RLVR.
Semantic-Space Exploration and Exploitation in RLVR for LLM Reasoning
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院,清华大学);University of California, Los Angeles(加州大学洛杉矶分校);Shenzhen Technology University(深圳技术大学)
From Exploration to Exploitation: A Two-Stage Entropy RLVR Approach for Noise-Tolerant MLLM Training
Independent Researcher(独立研究者);The Chinese University of Hong Kong(香港中文大学);City University of Hong Kong(香港城市大学);Hong Kong University of Science and Technology(香港科技大学);University of Hong Kong(香港大学)
Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging
Zhejiang University(浙江大学);Simon Fraser University(西蒙菲莎大学);The Chinese University of Hong Kong(香港中文大学);Zhejiang Key Lab of Accessible Perception and Intelligent Systems(浙江省可感知智能系统重点实验室)
Understanding Diversity Collapse in RLVR via the Lens of Overtraining
Sydney AI Centre, The University of Sydney(悉尼大学悉尼人工智能中心);Southeast University(东南大学);Microsoft(微软);Data61, CSIRO(澳大利亚联邦科学与工业研究组织Data61);Chongqing University(重庆大学);Nanyang Technological University(南洋理工大学)
The University of Texas at Austin(德克萨斯大学奥斯汀分校);UIUC(伊利诺伊大学香槟分校);Emory University(埃默里大学);Together AI(联合人工智能公司);Recursive Superintelligence Inc(递归超级智能公司);ELLIS Institute Tübingen(图宾根埃利斯研究所)
Beyond Forgetting: Diagnosing and Harnessing Shared Reasoning in Continual RLVR
State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室(BIGAI));Beijing Institute of Technology(北京理工大学);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);State Key Lab of General AI, School of Intelligence Science and Technology, Peking University(通用人工智能国家重点实验室、北京大学智能科学与技术学院)
Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
South China University of Technology(华南理工大学);Tsinghua University(清华大学);The Hong Kong University of Science and Technology(香港科技大学);Huawei 2012 Laboratories(华为2012实验室);Shenzhen Technology University(深圳技术大学)
Skip What You Can Predict: Predictive Repositioning for Policy Optimization for Efficient LLM Training
Bangladesh University of Engineering and Technology(孟加拉工程与技术大学);University of Maryland, College Park(马里兰大学学院公园分校);Illinois Institute of Technology(伊利诺伊理工学院);University of Central Florida(佛罗里达中央大学)
Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective
State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Zhejiang University(浙江大学);Tencent(腾讯);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院)
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR
School of Informatics, Xiamen University(厦门大学信息学院);LLM Team, Shopee Pte. Ltd.(Shopee 股份有限公司语言模型团队);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院)
Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);Huawei(华为)
When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVR
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University(国家多媒体软件工程研究中心,武汉大学计算机学院);Meituan Longcat Team(美团Longcat团队);The University of Sydney(悉尼大学);University of Science and Technology of China(中国科学技术大学)
Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards
Shanghai Innovation Institute, Shanghai, China(上海创新研究院);College of Future Information Technology, Fudan University, Shanghai, China(未来信息科技学院,复旦大学);Shanghai AI Laboratory, Shanghai, China(上海人工智能实验室);The Chinese University of Hong Kong, Hong Kong, China(香港中文大学)
Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR
New York University(纽约大学);University of Chicago(芝加哥大学);University of Waterloo(滑铁卢大学);University of Alberta(阿尔伯塔大学);NYU Shanghai(纽约大学上海分校);Alberta Machine Intelligence Institute (Amii)(阿尔伯塔机器智能研究所)
Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China(天津大学计算机科学与技术学院 TJUNLP 实验室,中国);Independent Researcher(独立研究者);College of Computer and Information Engineering, Tianjin Normal University, China(天津师范大学计算机与信息工程学院,中国)
HTPO: Towards Exploration-Exploitation Balanced Policy Optimization via Hierarchical Token-level Objective Control
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University(上海交通大学信息科学与电子工程学院);Xiaohongshu Inc.(小红书公司);MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University(上海交通大学人工智能研究院)
Segmental Advantage Estimation: Enhancing PPO for Long-Context LLM Training
LLM Department, Tencent(腾讯大语言模型部门);University of Chinese Academy of Sciences(中国科学院大学);Intelligent Software Research Center, Institute of Software, CAS(软件研究所智能软件研究中心);State Key Lab of Processors, Institute of Computing Technology, CAS(计算技术研究所处理器国家重点实验室)
Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
State Key Lab of Processors, Institute of Computing Technology, CAS(处理器国家重点实验室,计算技术研究所,中国科学院);University of Science and Technology of China(中国科学技术大学);University of Chinese Academy of Sciences(中国科学院大学);StepFun Inc(StepFun公司)