思考更长以探索更深入:通过长度激励强化学习学习上下文中的探索
Think Longer to Explore Deeper: Learn to Explore In-Context via Length-Incentivized Reinforcement Learning
AI总结:
通过长度激励强化学习,提升模型在上下文中的探索能力,从而在领域内和领域外任务中取得性能提升。
摘要:
实现有效的测试时间扩展需要模型具备上下文探索能力——内在的生成、验证和细化多个推理假设的能力。基于状态覆盖理论,我们的分析发现了一个关键瓶颈:虽然更广泛的状态覆盖需要更长的推理轨迹,但在自回归生成过程中,采样此类序列的概率呈指数衰减,我们称之为“浅层探索陷阱”。为弥合这一差距,我们提出了长度激励探索(方法)。这是一种简单而有效的配方,通过基于长度的奖励与冗余惩罚相结合,从而在两步过程中最大化状态覆盖。在不同模型(Qwen3、Llama)上的全面实验表明,方法有效激励了上下文探索。结果,我们的方法在领域内任务上平均提高了4.4%,在领域外基准测试上提高了2.7%。
英文摘要:
Achieving effective test-time scaling requires models to engage in In-Context Exploration -- the intrinsic ability to generate, verify, and refine multiple reasoning hypotheses within a single continuous context. Grounded in State Coverage theory, our analysis identifies a critical bottleneck to enabling this capability: while broader state coverage requires longer reasoning trajectories, the probability of sampling such sequences decays exponentially during autoregressive generation, a phenomenon we term the ``Shallow Exploration Trap''. To bridge this gap, we propose Length-Incentivized Exploration(\method). This simple yet effective recipe explicitly encourages models to explore more via a length-based reward coupled with a redundancy penalty, thereby maximizing state coverage in two-step manner. Comprehensive experiments across different models (Qwen3, Llama) demonstrate that \method effectively incentivize in-context exploration. As a result, our method achieves an average improvement of 4.4\% on in-domain tasks and a 2.7\% gain on out-of-domain benchmarks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。