arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11748cs.CL
收藏

思考更长以探索更深入:通过长度激励强化学习学习上下文中的探索

Think Longer to Explore Deeper: Learn to Explore In-Context via Length-Incentivized Reinforcement Learning

Zhejiang University(浙江大学);Westlake University(西湖大学);Shanghai AI Laboratory(上海人工智能实验室);Nanjing University(南京大学);Institute of Advanced Technology, Westlake Institute for Advanced Study(西湖研究所以高级技术研究所);The Chinese University of Hong Kong(香港中文大学)

Futing Wang, Jianhao Yan, Yun Luo, Ganqu Cui, Zhi Wang, Xiaoye Qu, Yue Zhang, Yu Cheng, Tao Lin

AI总结:

通过长度激励强化学习,提升模型在上下文中的探索能力,从而在领域内和领域外任务中取得性能提升。

摘要:

实现有效的测试时间扩展需要模型具备上下文探索能力——内在的生成、验证和细化多个推理假设的能力。基于状态覆盖理论,我们的分析发现了一个关键瓶颈:虽然更广泛的状态覆盖需要更长的推理轨迹,但在自回归生成过程中,采样此类序列的概率呈指数衰减,我们称之为“浅层探索陷阱”。为弥合这一差距,我们提出了长度激励探索(方法)。这是一种简单而有效的配方,通过基于长度的奖励与冗余惩罚相结合,从而在两步过程中最大化状态覆盖。在不同模型(Qwen3、Llama)上的全面实验表明,方法有效激励了上下文探索。结果,我们的方法在领域内任务上平均提高了4.4%,在领域外基准测试上提高了2.7%。

英文摘要:

Achieving effective test-time scaling requires models to engage in In-Context Exploration -- the intrinsic ability to generate, verify, and refine multiple reasoning hypotheses within a single continuous context. Grounded in State Coverage theory, our analysis identifies a critical bottleneck to enabling this capability: while broader state coverage requires longer reasoning trajectories, the probability of sampling such sequences decays exponentially during autoregressive generation, a phenomenon we term the ``Shallow Exploration Trap''. To bridge this gap, we propose Length-Incentivized Exploration(\method). This simple yet effective recipe explicitly encourages models to explore more via a length-based reward coupled with a redundancy penalty, thereby maximizing state coverage in two-step manner. Comprehensive experiments across different models (Qwen3, Llama) demonstrate that \method effectively incentivize in-context exploration. As a result, our method achieves an average improvement of 4.4\% on in-domain tasks and a 2.7\% gain on out-of-domain benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑