arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.01928cs.LGcs.AI
收藏

MALT:通过多智能体大语言模型训练提升推理能力

MALT: Improving Reasoning with Multi-Agent LLM Training

University of Oxford(牛津大学);Cooperative AI Foundation(合作人工智能基金会);MBZUAI(穆扎夫卡尔人工智能研究所);Stanford University(斯坦福大学)

Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Rafael Rafailov, Ivan Laptev, Philip H. S. Torr, Fabio Pizzati, Ronald Clark, Christian Schroeder de Witt

AI总结:

针对大语言模型单思维链推理的局限性,提出多智能体后训练策略MALT,通过异构智能体流水线和值迭代自动生成训练数据,在多个推理基准上实现显著性能提升。

摘要:

大语言模型(LLM)通常以单条思维链生成答案,这限制了它们在复杂任务中探索推理路径或对错误输出进行自我修正的能力。本文提出MALT(Multi-Agent LLM Training,多智能体大语言模型训练),这是一种新型后训练策略,采用由异构智能体组成的顺序流水线,将推理过程划分为生成、验证和细化步骤。在数据生成阶段,对每个智能体进行重复采样以构建多智能体搜索树,并依据真值数据对最终输出评分。随后我们应用值迭代将奖励信号反向传播至每个基于角色条件的模型,在无需人类或教师模型监督的情况下自动生成多智能体后训练数据。我们的离策略方法让每个智能体通过学习正确与错误轨迹实现专业化,最终优化端到端推理链。在MATH、GSM8K和CSQA基准上,MALT相较于相同基线LLM分别取得15.66%、7.42%和9.40%的相对提升,是多智能体协同训练方向的重要进展。

英文摘要:

Large Language Models (LLMs) often produce answers with a single chain-of-thought, which restricts their ability to explore reasoning paths or self-correct flawed outputs in complex tasks. In this paper, we introduce MALT (Multi-Agent LLM Training), a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps using a sequential pipeline of heterogeneous agents. During data generation, each agent is repeatedly sampled to form a multi-agent search tree, where final outputs are graded against ground-truth data. We then apply value iteration to propagate reward signals back to each role-conditioned model, automatically producing multi-agent post-training data without human or teacher-model supervision. Our off-policy approach allows each agent to specialize by learning from correct and incorrect trajectories, ultimately improving the end-to-end reasoning chain. On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with a relative improvement of 15.66%, 7.42%, and 9.40% respectively, making it an important advance towards multi-agent cooperative training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑