arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08344cs.AI
收藏

OPE:通过大纲引导路径探索克服并行思维中的信息饱和

OPE: Overcoming Information Saturation in Parallel Thinking via Outline-Guided Path Exploration

National Engineering Research Center for Software Engineering, Peking University, Beijing, China(软件工程国家工程研究中心,北京大学,北京,中国)

Qi Guo, Jianing Wang, Deyang Kong, Xiangyu Xi, Jianfei Zhang, Yi Lu, Jingang Wang, Wei Wang, Shikun Zhang, Wei Ye

AI总结:

OPE通过生成多样化的推理大纲,减少信息冗余,提升并行思维在复杂问题中的推理性能。

摘要:

并行思维已成为大型推理模型(LRMs)解决复杂问题的新范式。近期方法利用强化学习(RL)来增强并行思维,旨在解决监督微调在计算资源和有效性上的限制。然而,大多数现有研究主要集中在优化聚合阶段,对路径探索阶段的关注有限。在本文中,我们理论分析了在可验证奖励强化学习(RLVR)设定下并行思维的优化,并发现探索路径间的互信息瓶颈从根本上限制了整体性能。为解决这一问题,我们提出了大纲引导路径探索(OPE),通过在并行路径推理之前生成多样化的推理大纲,显式地将解空间划分为多个部分,从而减少信息冗余并提高探索路径中信息捕获的多样性。我们采用迭代RL策略实现OPE,该策略独立优化大纲规划和大纲引导的推理。在多个具有挑战性的数学基准测试中进行的广泛实验表明,OPE有效提升了不同聚合策略下的推理性能,使LRMs能够更可靠地发现正确解。

英文摘要:

Parallel thinking has emerged as a new paradigm for large reasoning models (LRMs) in tackling complex problems. Recent methods leverage Reinforcement Learning (RL) to enhance parallel thinking, aiming to address the limitations in computational resources and effectiveness encountered with supervised fine-tuning. However, most existing studies primarily focus on optimizing the aggregation phase, with limited attention to the path exploration stage. In this paper, we theoretically analyze the optimization of parallel thinking under the Reinforcement Learning with Verifiable Rewards (RLVR) setting, and identify that the mutual information bottleneck among exploration paths fundamentally restricts overall performance. To address this, we propose Outline-Guided Path Exploration (OPE), which explicitly partitions the solution space by generating diverse reasoning outlines prior to parallel path reasoning, thereby reducing information redundancy and improving the diversity of information captured across exploration paths. We implement OPE with an iterative RL strategy that optimizes outline planning and outline-guided reasoning independently. Extensive experiments across multiple challenging mathematical benchmarks demonstrate that OPE effectively improves reasoning performance in different aggregation strategies, enabling LRMs to more reliably discover correct solutions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑