Reinforcement learning with verifiable rewards (RLVR) has been a main driver of recent breakthroughs in large reasoning models. Yet it remains a mystery how rewards based solely on final outcomes can help overcome the long-horizon barrier to extended reasoning. To understand this, we develop a theory of the training dynamics of RLVR for transformers on compositional reasoning tasks. Our theory shows that mixed-difficulty training naturally induces an implicit curriculum: without any explicit schedule, easier problems become learnable first and shape the frontier for harder ones, creating a learning progression from easy to hard during optimization. The effectiveness of this curriculum is governed by the smoothness of the difficulty spectrum. When the spectrum is smooth, training dynamics enter a well-behaved relay regime, in which persistent gradient signals on easier problems make slightly harder ones tractable and keep training at the edge of competence. When the spectrum contains abrupt discontinuities, training undergoes grokking-type phase transitions with prolonged plateaus before progress recurs. As a technical contribution, our analysis develops and adapts techniques from Fourier analysis on finite groups to our setting. We validate the predicted mechanisms empirically via controlled synthetic experiments and real-model RLVR runs.
Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVR
Jinesis Lab, University of Toronto & Vector Institute（Jinesis实验室，多伦多大学与向量研究所）；Carnegie Mellon University（卡内基梅隆大学）；Princeton University（普林斯顿大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；ELLIS Institute Tübingen（ELLIS研究所图宾根）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；EuroSafeAI
Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculum
Seoul National University（首尔国立大学）；Artificial Intelligence Institute of Seoul National University (AIIS)（首尔国立大学人工智能研究所（AIIS））；Automation and Systems Research Institute (ASRI)（自动化与系统研究所（ASRI））
Towards Long-Horizon Vessel Trajectory and Destination Forecasting with Reasoning Large Language Models
Institute of High Performance Computing (IHPC), A*STAR, Singapore（新加坡科技研究局高性能计算研究所）；The Key Laboratory of Road and Traffic Engineering, Ministry of Education, Tongji University（同济大学道路与交通工程教育部重点实验室）；Meituan Inc., Shenzhen, China（美团（深圳））；Centre for Frontier AI Research (CFAR), A*STAR, Singapore（新加坡科技研究局前沿人工智能研究中心）；Nankai University（南开大学）；School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）
Scheduling Your LLM Reinforcement Learning with Reasoning Trees
Cranberry-Lemon University（Cranberry-Lemon 大学）；University of the Witwatersrand（独立研究者）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））
Implicit Language Models are RNNs: Balancing Parallelization and Expressivity
Chair of Highly-Parallel VLSI Systems and Neuro-Microelectronics , TUD Dresden University of Technology, Dresden, Germany（德累斯顿技术大学神经微电子学系）；Microsoft Research, Cambridge, United Kingdom（微软研究院）
Hierarchical Reinforcement Learning in StarCraft II with Human Expertise in Subgoals Selection
NUS（新加坡国立大学）；NTU（南洋理工大学）
这项工作的灵感来自于分层强化学习（HRL）（Barto and Mahadevan 2003; Hengst 2010）的最新进展，以及基于经验的子目标选择，经验重放（Lin 1993; Andrychowicz et al. 2017）和基于任务的课程学习（Bengio et al. 2009; Zaremba and Sutskever 2014）的学习效率提高。我们提出了一种新的方法来集成HRL，经验重放和有效的子目标选择，通过一个隐式的课程设计的基础上，人类的专业知识，以支持样本有效的学习和提高解释性的代理的行为。在许多领域，人类的专业知识仍然是不可或缺的，如医学（Buch，Ahmed和Maruthappu 2018）和法律（Cath 2018），出于道德和法律原因，可解释性，可解释性和透明度在决策过程中至关重要。我们的方法简化了复杂的任务集，通过将它们分解为不同抽象级别的子目标来实现总体目标。简化相关的主观知识也显著减少了用于RL探索的计算资源，特别是在高速，变化和复杂的环境中，过渡动态不能在短时间内有效地学习和建模。在两个星际争霸II（SC2）（Vinyals et al. 2017）小游戏中的实验结果表明，我们的方法可以实现比平面和端到端RL方法更好的样本效率，并为解释代理的性能提供了一种有效的方法。
Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；AMAP, Alibaba Group（阿里集团AMAP）；Xiamen University（厦门大学）；Dalian University of Technology（大连理工大学）
Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learning
School of Data Science, Fudan University（复旦大学数据科学学院）；University of Chinese Academy of Sciences（中国科学院大学）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）
Weak-to-Strong Generalization via Direct On-Policy Distillation
SIA-Lab of Tsinghua AIR and ByteDance Seed（清华-字节跳动联合研究中心SIA实验室）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Peking University（北京大学）
UR$^2$: Unify RAG and Reasoning through Reinforcement Learning
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China（计算机科学与技术系，人工智能研究院，清华大学，北京，中国）；Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China（人工智能产业研究机构（AIR），清华大学，北京，中国）；School of Management Science & Information Engineering, Hebei University of Economics and Business, Hebei, China（管理科学与信息工程学院，河北经贸大学，河北，中国）
SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning
National University of Singapore（新加坡国立大学）；Northeastern University（东北大学）；Sea AI Lab（Sea AI 实验室）；Centre for Frontier AI Research (CFAR), A*STAR（前沿人工智能研究中心（CFAR），A*STAR）；Plastic Labs；University of Washington（华盛顿大学）
Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence
The Hong Kong University of Science and Technology（香港科技大学）；Zhongguancun Academy（中关村学院）；Xi’an Jiaotong University（西安交通大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；Hong Kong Baptist University（香港浸会大学）；Hunyuan Tencent（腾讯混元）；National University of Singapore（新加坡国立大学）；Xiamen University（厦门大学）
