Reinforcement learning has become the standard for improving reasoning in large language models, yet evidence increasingly suggests that RL does not teach new strategies; it redistributes probability mass over solutions the base model already contains. In this work, we ask: if RL merely steers the model toward paths it already knows, is the RL optimization loop itself necessary? Through token-level analysis across multiple model families and RL algorithms, we find that RL's beneficial footprint is a sparse, predictable correction concentrated at high-entropy decision points where the model is uncertain which branch to take. Only 1--3\% of token positions are affected, the promoted token always lies within the base model's top-5 alternatives, and targeted corrections at those few positions causally recover a large fraction of RL's accuracy gain, while random corrections fail. The base model's own entropy identifies these positions without any RL-trained model, and the entire correction is low-dimensional, representable in a tiny fraction of model parameters. These findings reframe reasoning improvement as sparse policy selection, not capability acquisition. We translate this insight into ReasonMaxxer, a minimal RL-free method that applies contrastive loss only at entropy-gated decision points, using a few hundred base-model rollouts and no online generation. Across three model families, six scales, and six math reasoning benchmarks, ReasonMaxxer matches or exceeds full RL performance while requiring only tens of problems and minutes of single-GPU training, a reduction in training cost of roughly three orders of magnitude.
Rethinking On-policy Optimization for Query Augmentation
University of Utah(犹他大学);The University of Queensland(昆士兰大学);University of Waterloo(滑铁卢大学);New York University(纽约大学);University of Notre Dame(圣母大学);Université de Montréal(蒙特利尔大学);Google DeepMind(谷歌DeepMind);University of Oklahoma(俄克拉荷马大学)
Rethinking Continual Experience Internalization for Self-Evolving LLM Agents
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院 Gallagher 学院);School of Software, Beihang University(北航软件学院);Meituan(美团)
Rethinking On-Policy Distillation of Large Language Models II: One Training Example
University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);Northeastern University(东北大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Johns Hopkins University(约翰斯·霍普金斯大学)
How Brittle is Agent Safety? Rethinking Agent Risk under Intent Concealment and Task Complexity
School of Computer Science and Technology Xi’an Jiaotong University China(西安交通大学计算机科学与技术学院);Shanghai AI Laboratory(上海人工智能实验室);MOE KLINNS Lab Xi’an Jiaotong University China(西安交通大学教育部KLINNS实验室)
VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training
College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Honor Device Co., Ltd(荣耀设备有限公司);Institute of Trustworthy Embodied AI, Fudan University(复旦大学可信具身人工智能研究院);Shanghai Key Laboratory of Multimodal Embodied AI(上海多模态具身人工智能重点实验室)
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(人工智能通用智能国家重点实验室,智能科学与技术学院,北京大学);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);School of Economics, Peking University(经济学院,北京大学);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学)
Rethinking Length-Based Training: Batch Composition and Loss Normalization in Speech Token Language Models
Beijing Institute of Technology, Zhuhai(北京理工大学珠海校区);Institute of Science Tokyo(东京科学大学);Northeastern University(东北大学);Sichuan University(四川大学);Wuhan University(武汉大学);Ant Group(蚂蚁集团)
ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models
Huazhong University of Science and Technology(华中科技大学);Zhongguancun Academy(中关村学院);DeepCybo;Harbin Institute of Technology(哈尔滨工业大学);Beihang University(北京航空航天大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Zhengzhou University(郑州大学);Zhongguancun Institute of Artificial Intelligence(中关村人工智能研究院)