MAgICoRe：用于推理的多智能体、迭代式、由粗到细的精炼
MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning
AI总结：
提出MAgICoRe多智能体迭代精炼框架，通过问题难度分级、引入外部逐步奖励模型和多智能体协作循环，有效解决LLM推理中的过度精炼、错误定位不准和精炼不足问题，在5个数学数据集上显著优于现有基线。
摘要：
大型语言模型（LLM）的推理能力可以通过测试时聚合策略来提升，即生成多个样本并在生成的样本间进行投票。虽然这些策略能提高性能，但它们通常会达到饱和点。精炼提供了一种替代方案，通过使用LLM生成的反馈来提高解答质量。然而，精炼引入了3个关键挑战：（1）过度精炼：对所有实例进行统一精炼可能会导致过度修正，从而降低整体性能。（2）无法定位和解决错误：LLM的自我纠正能力有限，难以识别和纠正自身的错误。（3）精炼不足：决定需要多少次精炼迭代并非易事，过早停止可能会留下未解决的错误。为了解决这些问题，我们提出了MAgICoRe，它通过将问题难度分类为简单或困难来避免过度精炼，对简单问题使用粗粒度聚合，对困难问题使用细粒度且迭代式的多智能体精炼。为了提高错误定位能力，我们引入了外部的逐步奖励模型（RM）分数。此外，为了确保有效的精炼，我们采用了一个包含三个智能体的多智能体循环：Solver（求解器）、Reviewer（审查器，基于逐步RM分数生成针对性反馈）和Refiner（精炼器，整合反馈）。为了确保充分的精炼，我们重新评估更新后的解答，迭代地启动进一步的精炼轮次。我们在Llama-3-8B和GPT-3.5上评估了MAgICoRe，并展示了其在5个数学数据集上的有效性。即使仅进行一次迭代，MAgICoRe也优于Self-Consistency 3.4%、Best-of-k 3.2%和Self-Refine 4.0%，同时使用的样本不到一半。与基线的迭代精炼不同，MAgICoRe在更多迭代下能持续改进。最后，我们的消融实验突显了MAgICoRe的RM和多智能体通信的重要性。
英文摘要：
Large Language Models' (LLM) reasoning can be improved using test-time aggregation strategies, i.e., generating multiple samples and voting among generated samples. While these improve performance, they often reach a saturation point. Refinement offers an alternative by using LLM-generated feedback to improve solution quality. However, refinement introduces 3 key challenges: (1) Excessive refinement: Uniformly refining all instances can over-correct and reduce the overall performance. (2) Inability to localize and address errors: LLMs have a limited ability to self-correct and struggle to identify and correct their own mistakes. (3) Insufficient refinement: Deciding how many iterations of refinement are needed is non-trivial, and stopping too soon could leave errors unaddressed. To tackle these issues, we propose MAgICoRe, which avoids excessive refinement by categorizing problem difficulty as easy or hard, solving easy problems with coarse-grained aggregation and hard ones with fine-grained and iterative multi-agent refinement. To improve error localization, we incorporate external step-wise reward model (RM) scores. Moreover, to ensure effective refinement, we employ a multi-agent loop with three agents: Solver, Reviewer (which generates targeted feedback based on step-wise RM scores), and the Refiner (which incorporates feedback). To ensure sufficient refinement, we re-evaluate updated solutions, iteratively initiating further rounds of refinement. We evaluate MAgICoRe on Llama-3-8B and GPT-3.5 and show its effectiveness across 5 math datasets. Even one iteration of MAgICoRe beats Self-Consistency by 3.4%, Best-of-k by 3.2%, and Self-Refine by 4.0% while using less than half the samples. Unlike iterative refinement with baselines, MAgICoRe continues to improve with more iterations. Finally, our ablations highlight the importance of MAgICoRe's RMs and multi-agent communication.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
