我们提出一种旨在增强大型语言模型(LLM)推理能力的方法,该方法通过受 AlphaZero 成功策略启发的迭代偏好学习过程实现。我们的工作利用 Monte Carlo Tree Search(MCTS)迭代收集偏好数据,并借助其前瞻能力将实例级奖励分解为更细粒度的步骤级信号。为提升中间步骤的一致性,我们结合了结果验证和逐步自评估,持续更新对新生成数据的质量评估。所提出的算法使用 Direct Preference Optimization(DPO),利用这些新生成的步骤级偏好数据更新 LLM 策略。理论分析揭示了使用 on-policy 采样数据对于成功自我改进的重要性。在多种算术和常识推理任务上的广泛评估表明,相比现有模型,性能取得了显著提升。例如,我们的方法在 GSM8K、MATH 和 ARC-C 上优于 Mistral-7B Supervised Fine-Tuning(SFT)基线,准确率分别大幅提升至 81.8%(+5.9%)、34.7%(+5.8%)和 76.4%(+15.8%)。此外,我们的研究还深入探讨了训练与推理计算之间的权衡,为该方法如何有效最大化性能提升提供了见解。我们的代码已公开发布于 https://github.com/YuxiXie/MCTS-DPO。
英文摘要:
We introduce an approach aimed at enhancing the reasoning capabilities of Large Language Models (LLMs) through an iterative preference learning process inspired by the successful strategy employed by AlphaZero. Our work leverages Monte Carlo Tree Search (MCTS) to iteratively collect preference data, utilizing its look-ahead ability to break down instance-level rewards into more granular step-level signals. To enhance consistency in intermediate steps, we combine outcome validation and stepwise self-evaluation, continually updating the quality assessment of newly generated data. The proposed algorithm employs Direct Preference Optimization (DPO) to update the LLM policy using this newly generated step-level preference data. Theoretical analysis reveals the importance of using on-policy sampled data for successful self-improving. Extensive evaluations on various arithmetic and commonsense reasoning tasks demonstrate remarkable performance improvements over existing models. For instance, our approach outperforms the Mistral-7B Supervised Fine-Tuning (SFT) baseline on GSM8K, MATH, and ARC-C, with substantial increases in accuracy to $81.8\%$ (+$5.9\%$), $34.7\%$ (+$5.8\%$), and $76.4\%$ (+$15.8\%$), respectively. Additionally, our research delves into the training and inference compute tradeoff, providing insights into how our method effectively maximizes performance gains. Our code is publicly available at https://github.com/YuxiXie/MCTS-DPO.
Reinforcement Learning with Pairwise Preferences in Long-Term Decision Problems
School of Computer Science, McGill University, Montreal, Quebec, Canada(麦吉尔大学计算机科学学院);Mila - Quebec AI Institute, Montreal, Quebec, Canada(魁北克人工智能研究所);Department of Electrical Engineering, Stanford University, Stanford, California, USA(斯坦福大学电气工程系)
PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinking
Center for Computational Science and Engineering(计算科学与工程中心);Schmarzman College of Computing(施马茨曼计算学院);Laboratory for Atomistic and Molecular Mechanics (LAMM)(原子与分子力学实验室);Massachusetts Institute of Technology(麻省理工学院)
Learning Trajectory Preferences for Manipulators via Iterative Improvement
Cornell University(康奈尔大学)
我们考虑的问题,学习良好的轨迹操作任务。这是具有挑战性的,因为定义良好轨迹的标准因用户、任务和环境而异。在本文中,我们提出了一个共同主动的在线学习框架教机器人的偏好,其用户的对象操作任务。我们的方法的关键新颖性在于从用户期望的反馈的类型:人类用户不需要证明最佳轨迹作为训练数据,但只需要迭代地提供轨迹,稍微改善目前提出的轨迹系统。我们认为,这种共同主动的偏好反馈可以更容易地从用户比演示的最佳轨迹,这往往是具有挑战性的和非直观的提供高自由度操纵器。尽管如此,我们的算法的理论遗憾界匹配的最优轨迹算法的渐近速度。我们证明了我们的算法在各种杂货店结账任务上的可推广性,对他们来说,偏好不仅受到被操纵的对象的影响,而且还受到周围环境的影响。\footnote{For more details and a demonstration video, visit: \url{http://pr.cs.cornell.edu/coactive}}
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Salesforce AI Research(Salesforce AI研究院);The Hong Kong University of Science and Technology(香港科技大学)
DynamicGTR: Leveraging Graph Topology Representation Preferences to Boost VLM Capabilities on Graph QAs
Southern University of Science and Technology(南方科技大学);Hong Kong University of Science and Technology(香港理工大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Beihang University(北京航空航天大学)