Reinforcement learning with verifiable rewards (RLVR) substantially improves mathematical reasoning, but what it adds to the base model remains unclear. We investigate this through a sequence of generation-time interventions on Qwen3-4B-Base, measuring how much of its RL-trained counterpart's gain each intervention recovers on 120 AIME problems. Power sampling with the same 8,192-token output limit brings no gain beyond greedy decoding. Yet borrowing the RL model's token only where the two models' greedy choices differ, roughly 4% of generated positions, recovers 90.5% of the pass@1 gain, with similar recovery across three model pairs on MATH-500. We find that even a single opening token has a substantial effect: forcing the base model to begin with "Alright" recovers 35% of the RL gain. The opening alone is insufficient, however. Later replacements after the same opening add another 5.2 percentage points to pass@1, and the fixed rules we test for inserting reflective words such as "Wait" do not reproduce this benefit. Sparse replacement also retains much of the base model's diversity while staying close to the RL model at pass@1. At k=256, it solves 67.5% of the problems, significantly more than the RL model (61.4%) and a mixture of whole responses matched in pass@1 (61.45%). The comparison is at equal attempt counts rather than equal compute, since rl-replace consults both models at every decoding step. These results show that the opening can recover a meaningful part of the RL gain, but a larger contribution comes from later choices, which can be borrowed while retaining much of the base model's diversity. Code is available at https://github.com/CyrusZhang04/WhatRLBrings.
Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoning
School of Management, University of Science and Technology of China(中国科学技术大学管理学院);Department of Statistics, London School of Economics and Political Science(伦敦政治经济学院统计系);School of Mathematics, University of Birmingham(伯明翰大学数学学院);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);Department of Mathematics, Tsinghua University(清华大学数学系)
Self-Reasoning Language Models: Unfold Hidden Reasoning Chains with Few Reasoning Catalyst
The Chinese University of Hong Kong(香港中文大学);ByteDance(字节跳动);The University of Edinburgh(爱丁堡大学);Beihang University(北航);MoE Key Laboratory of High Confidence Software Technologies(高可信软件技术MoE实验室)
What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysis
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);Hong Kong University of Science and Technology(香港科技大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Evolutionary Pre-Prompt Optimization for Mathematical Reasoning
Meta AI Paris France(Meta AI);TAU, INRIA;LISN (CNRS \& Univ. Paris-Saclay) Orsay France;INSA Rouen Normandy, University of Rouen Normandy, LITIS UR 4108 Rouen France;Meta AI;LISN (CNRS \& Univ. Paris-Saclay);INSA Rouen Normandy, University of Rouen Normandy, LITIS UR 4108
MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China(认知与决策智能复杂系统重点实验室,自动化研究所,中国科学院,北京,中国);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);Tsinghua University(清华大学)
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University(上海数据科学关键实验室,计算机科学与人工智能学院,复旦大学);The Ohio State University(俄亥俄州立大学);Shanghai Academy of AI for Science(上海人工智能科学研究院)