探索与利用：通过截断、熵和虚假奖励重新思考RLVR
Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
AI总结：
本文通过截断、熵和虚假奖励机制，重新审视RLVR框架，揭示了探索与利用权衡对大语言模型推理能力提升的影响。
摘要：
本文探讨了强化学习中可验证奖励（RLVR）框架下的探索-利用权衡问题，该框架旨在提升大语言模型（LLMs）的推理能力。近期研究表明，RLVR通过两种看似矛盾的机制能够促使LLMs产生强大的数学推理能力：虚假奖励，通过奖励与真实结果无关的成果来抑制利用，以及熵最小化，通过推动模型向更自信和确定性的输出发展来抑制探索，揭示了一个令人困惑的动态：同时抑制利用和探索的机制反而能提升推理性能，但这些效果背后的原理仍不明确。本文聚焦两个基本问题：（i）策略熵与性能之间的关系，以及（ii）虚假奖励是否能带来收益，可能通过截断偏差与模型污染的相互作用。我们的研究结果表明，虚假奖励下的截断偏差会降低策略熵，导致更自信和确定性的输出，而单纯的熵最小化不足以带来提升。我们进一步提出了一个奖励不匹配模型，解释了为何虚假奖励能够在污染环境下提升性能。我们的发现澄清了虚假奖励收益背后的机制，并为更有效的RLVR训练提供了原理。
英文摘要：
This paper examines the exploration-exploitation trade-off in reinforcement learning with verifiable rewards (RLVR), a framework for improving the reasoning of Large Language Models (LLMs). Recent studies suggest that RLVR can elicit strong mathematical reasoning in LLMs through two seemingly paradoxical mechanisms: spurious rewards, which suppress exploitation by rewarding outcomes unrelated to the ground truth, and entropy minimization, which suppresses exploration by pushing the model toward more confident and deterministic outputs, highlighting a puzzling dynamic: both discouraging exploitation and discouraging exploration improve reasoning performance, yet the underlying principles that reconcile these effects remain poorly understood. We focus on two fundamental questions: (i) how policy entropy relates to performance, and (ii) whether spurious rewards yield gains, potentially through the interplay of clipping bias and model contamination. Our results show that clipping bias under spurious rewards reduces policy entropy, leading to more confident and deterministic outputs, while entropy minimization alone is insufficient for improvement. We further propose a reward-misalignment model explaining why spurious rewards can enhance performance beyond contaminated settings. Our findings clarify the mechanisms behind spurious-reward benefits and provide principles for more effective RLVR training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
