Linear Dynamics in the RLVR Training of Large Language Models
Department of Data Science, City University of Hong Kong(香港城市大学数据科学系);Hong Kong Institute of AI for Science, City University of Hong Kong(香港城市大学人工智能科学研究院);Li Auto Inc.;Beihang University(北航大学)
Major revision: substantially reorganized the manuscript and added a theoretical explanation section. The replacement is intended for the same arXiv paper; the core topic and contribution remain the same
Reinforcement learning with verifiable rewards (RLVR) has driven significant performance gains in reasoning-oriented large language models (LLMs), yet its internal training dynamics remain largely a black box. In this work, we perform a comprehensive trajectory-level analysis of RLVR and uncover a striking regularity: across various model families, RL algorithms, and training configurations, RLVR consistently enters a robust linear regime, where both parameter weights and output log-probabilities, measured rigorously via teacher-forced evaluation, evolve in a highly linear manner ($R^2 > 0.7$). Through controlled experiments and theoretical analysis, we demonstrate that this linearity is not a coincidence, but stems from the high-variance, noisy nature of RLVR training signals, which act as a low-pass filter to concentrate optimization along a stable, low-dimensional drift. Moreover, we show that this linear structure is not merely descriptive but powerfully predictive and actionable. Specifically, weight-space extrapolation matches the performance of standard RL optimization while achieving a 6.1x training speedup through periodic re-grounding. Meanwhile, output-space extrapolation serves as a lightweight intervention that effectively bypasses late-stage model collapse, consistently outperforming standard RL across mathematical and coding benchmarks, with an average performance improvement of 4.2%. Our code is available at https://github.com/Miaow-Lab/RLVR-Linearity.
RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution
National Key Laboratory for Novel Software Technology(新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);Huawei Technologies Co., Ltd.(华为技术有限公司)
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
School of Mathematical Sciences, Shanghai Jiao Tong University(上海交通大学数学科学学院);Institute of Natural Sciences, Shanghai Jiao Tong University(上海交通大学自然科学研究院)
Generalization of RLVR Using Causal Reasoning as a Testbed
Johns Hopkins University(约翰霍普金斯大学);University of Maryland, College Park(马里兰大学学院公园分校);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Microsoft Research Asia(微软亚洲研究院);Toyota Technological Institute at Chicago(芝加哥丰田技术研究所)
CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR
Qwen Large Model Application Team, Alibaba Institute of Automation, Chinese Academy of Sciences(文勤大模型应用团队,阿里巴巴自动化研究所,中国科学院);Qwen Large Model Application Team, Alibaba(文勤大模型应用团队,阿里巴巴)
Reinforcement Learning for Reasoning in Large Language Models with One Training Example
University of Washington(华盛顿大学);University of Southern California(南加州大学);Microsoft(微软);University of California, Santa Cruz(加州大学圣克鲁兹分校);Georgia Institute of Technology(佐治亚理工学院)
CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
Tencent AI Lab(腾讯AI实验室);Tencent Multimodal Department(腾讯多模态部门);University of North Carolina at Chapel Hill(北卡罗来纳大学夏洛特分校);University of Virginia(弗吉尼亚大学);University of Maryland, College Park(马里兰大学学院公园分校)
Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);Mila - Quebec AI Institute(Mila - 魁北克人工智能研究所);University of Waterloo(滑铁卢大学);The Chinese University of Hong Kong(香港中文大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)
Decouple before Integration: Test-time Synthesis of SFT and RLVR Task Vectors
Department of Systems Engineering and Engineering Management, Chinese University of Hong Kong, Hong Kong, China(系统工程与工程管理系,香港中文大学,香港,中国);Department of Computer Science, Hong Kong Baptist University, Hong Kong, China(计算机科学系,香港 Baptist 大学,香港,中国);DAMO Academy, Alibaba Group, Hangzhou, China(达摩院,阿里巴巴集团,杭州,中国);Hupan Lab, Hangzhou, China(虎派实验室,杭州,中国)
Does LLM Alignment Really Need Diversity? An Empirical Study of Adapting RLVR Methods for Moral Reasoning
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Microsoft Research(微软研究院);University of Michigan(密歇根大学);Shanghai Jiao Tong University(上海交通大学);CUHKSZ(香港中文大学(深圳));THU(清华大学)
MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning
Brown University(布朗大学);Salesforce AI Research(Salesforce AI研究院);NVIDIA Research(英伟达研究院);Carnegie Mellon University(卡内基梅隆大学);Princeton University(普林斯顿大学);Texas A&M University(德克萨斯A&M大学);California Institute of Technology(加州理工学院);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室)
Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China(天津大学计算机科学与技术学院 TJUNLP 实验室,中国);Independent Researcher(独立研究者);College of Computer and Information Engineering, Tianjin Normal University, China(天津师范大学计算机与信息工程学院,中国)
Combinatorial Synthesis: Scaling Code RLVR via Atomic Decomposition and Recombination
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学);Lero the Research Ireland Centre for Software, University of Limerick(利默尼克大学爱尔兰软件研究中心);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));The Hong Kong University of Science and Technology(香港科技大学)
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);National Engineering Research Center of New Electronic Publishing Technologies(国家新型电子出版技术工程研究中心)
A Verifier-Guided Explainable Reasoning Framework with Gold-Anchored QLoRA, Task-Aware Mixture-of-Experts, and Group-Relative RLVR
University of Information Technology (UIT)(信息科技大学);Ho Chi Minh City University of Technology (HCMUT)(胡志明市理工大学);Vietnam National University, Ho Chi Minh City(越南国家大学胡志明市分校);University of Economics Ho Chi Minh City (UEH)(胡志明市经济大学)
New Skills or Sharper Primitives? A Probabilistic Perspective on the Emergence of Reasoning in RLVR
University of Science(科学技术大学);The Chinese University of Hong Kong(香港中文大学);Peking University(北京大学);Nanjing University(南京大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学)
The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
College of Engineering and Computer Science, VinUniversity(Vin大学工程与计算机科学学院);Center for Environmental Intelligence, VinUniversity(Vin大学环境智能中心);University of Stuttgart(斯图加特大学);University of Notre Dame(诺特难大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);Max Planck Research School for Intelligent Systems (IMPRS-IS)(马克斯·普朗克智能系统研究学校)