arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.16614cs.AI
收藏

计数计数:基于计数的内在奖励激励大语言模型推理中的探索

Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards

Fudan University(复旦大学);Shanghai Innovation Institute(上海创新研究院);Tencent Youtu Lab(腾讯优图实验室)

Xuan Zhang, Ruixiao Li, Zhijian Zhou, Long Li, Yulei Qin, Ke Li, Xing Sun, Xiaoyu Tan, Chao Qu, Yuan Qi

AI总结:

本文提出MERCI算法,利用基于计数的内在奖励(通过轻量级CFN估计伪计数与不确定性)增强LLM推理的探索,集成于GRPO框架,在复杂推理基准上显著提升性能并促进多样化思维链。

摘要:

强化学习(RL)已成为增强大语言模型(LLMs)多步推理能力的一种引人注目的方式。然而,普遍的RL范式仍然依赖于稀疏的基于结果的奖励和有限的探索,这常常驱使LLMs走向重复且次优的推理模式。在本文中,我们研究了如何为LLM推理设计探索这一核心问题,并引入了MERCI(基于计数的内在奖励激励LLM推理中的探索),一种新颖的RL算法,通过原则性的内在奖励增强策略优化。基于计数探索的思想,MERCI利用轻量级的抛硬币网络(CFN)来估计推理轨迹上的伪计数及进一步的认知不确定性,并将其转化为一种重视新颖性同时保留任务奖励学习信号的内在奖励。我们将MERCI集成到一些先进的RL框架中,如组相对策略优化(GRPO)。在复杂推理基准上的实验表明,MERCI鼓励更丰富和更多样化的思维链,显著提高了相对于强基线的性能,并帮助策略摆脱局部常规以发现更好的解决方案。这表明我们针对性的内在动机可以使探索对语言模型推理可靠。

英文摘要:

Reinforcement Learning (RL) has become a compelling way to strengthen the multi step reasoning ability of Large Language Models (LLMs). However, prevalent RL paradigms still lean on sparse outcome-based rewards and limited exploration, which often drives LLMs toward repetitive and suboptimal reasoning patterns. In this paper, we study the central question of how to design exploration for LLM reasoning and introduce MERCI (Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards), a novel RL algorithm that augments policy optimization with a principled intrinsic reward. Building on the idea of count-based exploration, MERCI leverages a lightweight Coin Flipping Network (CFN) to estimate the pseudo count and further epistemic uncertainty over reasoning trajectories, and converts them into an intrinsic reward that values novelty while preserving the learning signal from task rewards. We integrate MERCI into some advanced RL frameworks like Group Relative Policy Optimization (GRPO). Experiments on complex reasoning benchmarks demonstrate that MERCI encourages richer and more varied chains of thought, significantly improves performance over strong baselines, and helps the policy escape local routines to discover better solutions. It indicates that our targeted intrinsic motivation can make exploration reliable for language model reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑