arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.00451cs.AIcs.LG
收藏

Monte Carlo Tree Search 通过迭代偏好学习增强推理能力

Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning

National University of Singapore(新加坡国立大学);Google DeepMind(谷歌DeepMind)

Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P. Lillicrap, Kenji Kawaguchi, Michael Shieh

AI总结:

本文提出 MCTS-DPO,利用 MCTS 将奖励分解为步骤级偏好信号,并通过 DPO 迭代优化 LLM,在多个算术与常识推理基准上显著提升准确率。

摘要:

我们提出一种旨在增强大型语言模型(LLM)推理能力的方法,该方法通过受 AlphaZero 成功策略启发的迭代偏好学习过程实现。我们的工作利用 Monte Carlo Tree Search(MCTS)迭代收集偏好数据,并借助其前瞻能力将实例级奖励分解为更细粒度的步骤级信号。为提升中间步骤的一致性,我们结合了结果验证和逐步自评估,持续更新对新生成数据的质量评估。所提出的算法使用 Direct Preference Optimization(DPO),利用这些新生成的步骤级偏好数据更新 LLM 策略。理论分析揭示了使用 on-policy 采样数据对于成功自我改进的重要性。在多种算术和常识推理任务上的广泛评估表明,相比现有模型,性能取得了显著提升。例如,我们的方法在 GSM8K、MATH 和 ARC-C 上优于 Mistral-7B Supervised Fine-Tuning(SFT)基线,准确率分别大幅提升至 81.8%(+5.9%)、34.7%(+5.8%)和 76.4%(+15.8%)。此外,我们的研究还深入探讨了训练与推理计算之间的权衡,为该方法如何有效最大化性能提升提供了见解。我们的代码已公开发布于 https://github.com/YuxiXie/MCTS-DPO。

英文摘要:

We introduce an approach aimed at enhancing the reasoning capabilities of Large Language Models (LLMs) through an iterative preference learning process inspired by the successful strategy employed by AlphaZero. Our work leverages Monte Carlo Tree Search (MCTS) to iteratively collect preference data, utilizing its look-ahead ability to break down instance-level rewards into more granular step-level signals. To enhance consistency in intermediate steps, we combine outcome validation and stepwise self-evaluation, continually updating the quality assessment of newly generated data. The proposed algorithm employs Direct Preference Optimization (DPO) to update the LLM policy using this newly generated step-level preference data. Theoretical analysis reveals the importance of using on-policy sampled data for successful self-improving. Extensive evaluations on various arithmetic and commonsense reasoning tasks demonstrate remarkable performance improvements over existing models. For instance, our approach outperforms the Mistral-7B Supervised Fine-Tuning (SFT) baseline on GSM8K, MATH, and ARC-C, with substantial increases in accuracy to $81.8\%$ (+$5.9\%$), $34.7\%$ (+$5.8\%$), and $76.4\%$ (+$15.8\%$), respectively. Additionally, our research delves into the training and inference compute tradeoff, providing insights into how our method effectively maximizes performance gains. Our code is publicly available at https://github.com/YuxiXie/MCTS-DPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑