arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.14917cs.CLcs.AI
收藏

BFS-PO:用于大规模推理模型的最佳优先搜索

BFS-PO: Best-First Search for Large Reasoning Models

Department of Engineering, University of Modena(Modena大学工程系);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学北京校区人工智能学院)

Fiorenzo Parascandolo, Wenhui Tan, Enver Sangineto, Xiaoyi Yu, Ruihua Song, Rita Cucchiara

AI总结:

BFS-PO是一种通过最佳优先搜索策略减少大规模推理模型过度思考问题的强化学习算法,提高了推理准确性和响应简洁性。

摘要:

大规模推理模型(LRMs)如OpenAI o1和DeepSeek-R1在使用长推理链进行推理任务时表现出色。然而,这同时也导致了计算成本的显著增加和生成冗长输出的现象,即过度思考。这种过度思考的倾向往往被强化学习(RL)算法如GRPO/DAPO加剧。在本文中,我们提出了BFS-PO,一种RL算法,通过最佳优先搜索探索策略来缓解这一问题。具体而言,BFS-PO通过基于最大熵节点的回溯机制寻找最短的正确答案。在训练过程中,BFS-PO生成逐步更短的响应,从而学习生成简洁的推理链。通过不同的基准测试和基础LRMs,我们展示了BFS-PO可以同时提高LRM的准确性和缩短其答案。

英文摘要:

Large Reasoning Models (LRMs) have shown excellent performance in reasoning tasks using long Chains of Thought (CoTs). However, this has also led to a significant increase of computational costs and the generation of verbose output, a phenomenon known as overthinking. The tendency to overthink is often exacerbated by Reinforcement Learning (RL) algorithms such as GRPO/DAPO. In this paper, we propose BFS-PO, an RL algorithm which alleviates this problem using a Best-First Search exploration strategy. Specifically, the reasoning chains generated at training time by BFS-PO are organized into a search tree, within which the shortest correct sequence is selected as the best and expanded using a backtracking mechanism based on maximum entropy nodes. In this way, we bias the exploration of the solution space towards the search for increasingly shorter solutions, training the LRM to generate progressively more concise answers. Using different benchmarks and base LRMs, we show that BFS-PO can simultaneously increase the LRM accuracy and shorten its reasoning chains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑