Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts
School of Data Science, Fudan University(复旦大学数据科学学院);Shanghai Institute of Artificial Intelligence for Education, East China Normal University(华东师范大学教育人工智能研究所);College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Ant Group(蚂蚁集团)
Reinforcement Learning (RL) has become a key driver for enhancing the long chain-of-thought (CoT) reasoning capabilities of Large Language Models (LLMs). However, prevalent methods like GRPO often fail when task difficulty exceeds model capacity, leading to reward sparsity and inefficient training. Prior work attempts to mitigate this with off-policy data, but such methods often induce severe distributional mismatches that destabilize policy updates. In this work, we identify a core issue underlying these failures, which we term low training affinity, and introduce Affinity, the first quantitative metric for monitoring the compatibility between external guidance and the model's intrinsic policy. To address this, we propose HINT, an adaptive framework designed to enhance reasoning capabilities while explicitly preserving high Affinity. First, instead of revealing partial answers, HINT supplies Meta-Hints, which act as abstract cognitive scaffolding to guide the model in articulating solutions independently. Second, to ensure stability, we integrate Affinity-Aware Policy Optimization (AAPO), which dynamically modulates the learning objective based on the Affinity. Extensive experiments across diverse benchmarks demonstrate that HINT consistently outperforms strong baselines, while exhibiting superior stability and robust generalization to out-of-distribution tasks. Code is available at https://github.com/ViviqwerAsd/HINT.
ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward
LMU Munich(慕尼黑大学);Harvard University(哈佛大学);University of Cambridge(剑桥大学);Mina AI;Konrad Zuse School of Excellence in Reliable AI (relAI)(康拉德·楚泽可靠人工智能卓越学校(relAI))
Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards
Tsinghua University(清华大学);Renmin University of China(中国人民大学);Institute of Information Engineering, CAS(中国科学院信息工程研究所);USTC(中国科学技术大学);The Australian National University(澳大利亚国立大学);Peking University(北京大学);University of Macau(澳门大学)
Weak-to-Strong Generalization via Direct On-Policy Distillation
SIA-Lab of Tsinghua AIR and ByteDance Seed(清华-字节跳动联合研究中心SIA实验室);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Peking University(北京大学)