Large Reasoning Models (LRMs) have shown excellent performance in reasoning tasks using long Chains of Thought (CoTs). However, this has also led to a significant increase of computational costs and the generation of verbose output, a phenomenon known as overthinking. The tendency to overthink is often exacerbated by Reinforcement Learning (RL) algorithms such as GRPO/DAPO. In this paper, we propose BFS-PO, an RL algorithm which alleviates this problem using a Best-First Search exploration strategy. Specifically, the reasoning chains generated at training time by BFS-PO are organized into a search tree, within which the shortest correct sequence is selected as the best and expanded using a backtracking mechanism based on maximum entropy nodes. In this way, we bias the exploration of the solution space towards the search for increasingly shorter solutions, training the LRM to generate progressively more concise answers. Using different benchmarks and base LRMs, we show that BFS-PO can simultaneously increase the LRM accuracy and shorten its reasoning chains.
FoE: Forest of Errors Makes the First Solution the Best in Large Reasoning Models
School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院通用人工智能国家重点实验室)
Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Process
Beihang University, Beijing, China(北京航空航天大学);Zhongguancun Laboratory, Beijing, China(中关村实验室);Beijing University of Posts and Telecommunications(北京邮电大学);Hong Kong University of Science and Technology(香港科学与技术大学);Xi'an Jiaotong University, Xi'an, China(西安交通大学);Tsinghua University, Beijing, China(清华大学);Singapore University of Technology and Design(新加坡科技设计大学)
TrajSelector: Harnessing Latent Representations for Efficient and Effective Best-of-N in Large Reasoning Model
Harbin Institute of Technology(哈尔滨工业大学);Zhongguancun Academy(中关村学院);Zhongguancun Institute of Artificial Intelligence(中关村人工智能研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Huazhong University of Science and Technology(华中科技大学);East China Normal University(东华大学)