Optima：优化基于 LLM 的多智能体系统的有效性与效率
Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System
AI总结：
本文提出 Optima 框架，通过生成、排序、选择与训练的迭代流程及 MCTS 式 DPO 数据生成优化多智能体通信，在 Llama 3 8B 上以极少 token 获得最高 2.8 倍性能提升。
摘要：
基于大语言模型（LLM）的多智能体系统（MAS）在协作式问题解决中展现出显著潜力，但仍面临关键挑战：通信效率低、可扩展性差，以及缺乏有效的参数更新优化方法。我们提出 Optima，这是一个新颖框架，通过 LLM 训练显著增强基于 LLM 的 MAS 中的通信效率和任务有效性，从而解决这些问题。Optima 采用迭代式生成、排序、选择和训练范式，并使用平衡任务性能、token 效率与通信可读性的奖励函数。我们探索了多种 RL 算法，包括 Supervised Fine-Tuning、Direct Preference Optimization 及其混合方法，为它们在有效性—效率之间的权衡提供了见解。我们将受 Monte Carlo Tree Search 启发的技术集成到 DPO 数据生成中，把对话轮次视为树节点，以探索多样的交互路径。在常见多智能体任务（包括信息不对称问答和复杂推理）上的评估表明，相较于基于 Llama 3 8B 的单智能体基线和原始 MAS，Optima 取得了一致且显著的改进；在需要大量信息交换的任务上，其使用不到 10% 的 token 即实现最高 2.8 倍的性能提升。此外，Optima 的效率提升为更有效地利用推理算力开辟了新的可能性，从而改善推理时扩展定律。通过解决基于 LLM 的 MAS 中的基础性挑战，Optima 展现了迈向可扩展、高效且有效的 MAS 的潜力（https://chenweize1998.github.io/optima-project-page）。
英文摘要：
Large Language Model (LLM) based multi-agent systems (MAS) show remarkable potential in collaborative problem-solving, yet they still face critical challenges: low communication efficiency, poor scalability, and a lack of effective parameter-updating optimization methods. We present Optima, a novel framework that addresses these issues by significantly enhancing both communication efficiency and task effectiveness in LLM-based MAS through LLM training. Optima employs an iterative generate, rank, select, and train paradigm with a reward function balancing task performance, token efficiency, and communication readability. We explore various RL algorithms, including Supervised Fine-Tuning, Direct Preference Optimization, and their hybrid approaches, providing insights into their effectiveness-efficiency trade-offs. We integrate Monte Carlo Tree Search-inspired techniques for DPO data generation, treating conversation turns as tree nodes to explore diverse interaction paths. Evaluated on common multi-agent tasks, including information-asymmetric question answering and complex reasoning, Optima shows consistent and substantial improvements over single-agent baselines and vanilla MAS based on Llama 3 8B, achieving up to 2.8x performance gain with less than 10\% tokens on tasks requiring heavy information exchange. Moreover, Optima's efficiency gains open new possibilities for leveraging inference-compute more effectively, leading to improved inference-time scaling laws. By addressing fundamental challenges in LLM-based MAS, Optima shows the potential towards scalable, efficient, and effective MAS (https://chenweize1998.github.io/optima-project-page).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
