arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.03128cs.AI
收藏

理解多智能体大语言模型框架:一个统一的基准测试和实验分析

Architectural Design, Not Only Model Intelligence, Governs Multi-Agent LLM Performance

Concordia University(康科迪亚大学)

Abdelghny Orogat, Ana Rostam, Essam Mansour

AI总结:

本文提出MAFBench统一评估套件,系统比较多智能体LLM框架,揭示架构设计对性能的显著影响。

摘要:

多智能体大语言模型(LLM)框架被广泛用于加速由大语言模型(LLM)驱动的智能体系统的发展。这些框架 impose 不同的架构结构,这些结构决定了智能体如何交互、存储信息以及协调任务。然而,这些框架对系统性能的影响仍不明确。这个空白至关重要,因为架构选择本身可以导致延迟和吞吐量的量级差异,以及显著的准确性和可扩展性变化。解决这一挑战需要(i)联合评估多个能力,如协调开销、内存行为、规划、专业化和协调,以及(ii)在受控的框架层面条件下进行这些评估以隔离架构效应。现有的基准测试专注于单个能力,并缺乏标准化的框架层面评估。我们通过(i)引入一个架构分类法,以系统地比较多智能体LLM框架的基本维度,以及(ii)开发MAFBench,一个统一的评估套件,将现有基准测试整合到标准化的执行管道中,来解决这些限制。使用MAFBench,我们对几个广泛使用的框架进行了受控的实证研究。我们的结果表明,框架层面的设计选择可以将延迟增加超过100倍,将规划准确性降低高达30%,并将协调成功率从超过90%降低到低于30%。最后,我们将发现转化为具体的架构设计原则和框架选择指导,并概述有前途的未来研究方向。

英文摘要:

Multi-agent LLM frameworks are data-intensive systems that govern how agents orchestrate tasks, manage state, and coordinate decisions. These architectural choices control execution overhead, memory behavior, planning effectiveness, and coordination scalability. Their impact on system performance remains poorly understood. Existing benchmarks evaluate individual agent capabilities in isolation and lack standardized framework-level comparison. We make four contributions. We introduce an architectural taxonomy that decomposes multi-agent LLM frameworks along five dimensions: orchestration, memory, planning interfaces, specialization, and communication topology. We develop MAFBench, a unified evaluation suite that integrates existing benchmarks within a standardized execution pipeline. We conduct a controlled empirical study across nine frameworks, fixing the underlying LLM and varying only architectural design choices. We distill the results into six evidence-based design principles. Architectural design, not only model intelligence, governs performance. Orchestration alone increases latency by over 60x, and a minimal implementation of the same paradigm isolates that cost as implementation rather than paradigm. Schema-constrained planning interfaces reduce accuracy by up to 32 points through formatting failures, not reasoning errors. Communication topology drops coordination success from above 90% to below 30% under mismatched structure. Memory architecture controls recall and scalability independent of context window size, and no evaluated framework natively supports controlled knowledge revision.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑