arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.09388cs.LGcs.CL
收藏

不要告诉答案,真正引导RL Rollout期间的推理

Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts

School of Data Science, Fudan University(复旦大学数据科学学院);Shanghai Institute of Artificial Intelligence for Education, East China Normal University(华东师范大学教育人工智能研究所);College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Ant Group(蚂蚁集团)

Xinyi Wang, Jinyi Han, Zishang Jiang, Tingyun Li, Jiaqing Liang, Sihang Jiang, Zhaoqian Dai, Shuguang Ma, Fei Yu, Yanghua Xiao

AI总结:

针对强化学习中任务难度超出模型能力导致奖励稀疏的问题,提出HINT框架,通过元提示和亲和力感知策略优化,提升模型推理能力并保持训练稳定性。

摘要:

强化学习已成为增强大型语言模型长链推理能力的关键驱动力。然而,像GRPO这样的流行方法在任务难度超过模型能力时常常失败,导致奖励稀疏和训练效率低下。先前的工作尝试通过离策略数据缓解这一问题,但此类方法往往引发严重的分布不匹配,从而破坏策略更新的稳定性。在这项工作中,我们识别了这些失败背后的核心问题,称为低训练亲和力,并引入了Affinity,这是第一个用于监控外部引导与模型内在策略之间兼容性的定量指标。为了解决这个问题,我们提出了HINT,一个自适应框架,旨在增强推理能力同时明确保持高亲和力。首先,HINT不揭示部分答案,而是提供元提示,作为抽象的认知支架,引导模型独立阐述解决方案。其次,为了确保稳定性,我们集成了亲和力感知策略优化,该优化根据亲和力动态调整学习目标。跨多个基准的大量实验表明,HINT始终优于强基线,同时在分布外任务中表现出卓越的稳定性和鲁棒泛化能力。代码可在该https URL获取。

英文摘要:

Reinforcement Learning (RL) has become a key driver for enhancing the long chain-of-thought (CoT) reasoning capabilities of Large Language Models (LLMs). However, prevalent methods like GRPO often fail when task difficulty exceeds model capacity, leading to reward sparsity and inefficient training. Prior work attempts to mitigate this with off-policy data, but such methods often induce severe distributional mismatches that destabilize policy updates. In this work, we identify a core issue underlying these failures, which we term low training affinity, and introduce Affinity, the first quantitative metric for monitoring the compatibility between external guidance and the model's intrinsic policy. To address this, we propose HINT, an adaptive framework designed to enhance reasoning capabilities while explicitly preserving high Affinity. First, instead of revealing partial answers, HINT supplies Meta-Hints, which act as abstract cognitive scaffolding to guide the model in articulating solutions independently. Second, to ensure stability, we integrate Affinity-Aware Policy Optimization (AAPO), which dynamically modulates the learning objective based on the Affinity. Extensive experiments across diverse benchmarks demonstrate that HINT consistently outperforms strong baselines, while exhibiting superior stability and robust generalization to out-of-distribution tasks. Code is available at https://github.com/ViviqwerAsd/HINT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑