arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.08835cs.AIcs.CLcs.LG
收藏

MASEval: 从模型到系统的多智能体评估扩展

MASEval: Extending Multi-Agent Evaluation from Models to Systems

Parameter Lab(参数实验室);University of Oxford(牛津大学);University of Tübingen(图宾根大学);TU Darmstadt(德累斯顿理工大学);MBZUAI;NAVER AI Lab(NAVER AI实验室);KAIST(韩国科学技术院)

Cornelius Emde, Alexander Rubinstein, Anmol Goel, Ahmed Heakl, Sangdoo Yun, Seong Joon Oh, Martin Gubri

AI总结:

MASEval通过系统层面的比较揭示框架选择与模型选择同等重要,为智能体系统设计提供新途径。

摘要:

LLM驱动的智能体系统快速普及,催生了丰富的框架生态系统(如smolagents、LangGraph、AutoGen、CAMEL、LlamaIndex等)。然而现有基准测试过于模型导向:它们固定智能体设置,不比较其他系统组件。我们主张实现决策显著影响性能,包括拓扑结构、编排逻辑和错误处理等选择。MASEval通过一个框架无关的库来填补这一评估空白,将整个系统作为分析单位。通过系统层面的系统比较,涵盖3个基准、3个模型和3个框架,我们发现框架选择与模型选择同样重要。MASEval使研究人员能够探索智能体系统的所有组件,开启系统设计的新途径,使从业者能够为特定应用场景找到最佳实现。MASEval在MIT许可下可用 https://github.com/parameterlab/MASEval。

英文摘要:

The rapid adoption of LLM-based agentic systems has produced a rich ecosystem of frameworks (smolagents, LangGraph, AutoGen, CAMEL, LlamaIndex, i.a.). Yet existing benchmarks are model-centric: they fix the agentic setup and do not compare other system components. We argue that implementation decisions substantially impact performance, including choices such as topology, orchestration logic, and error handling. MASEval addresses this evaluation gap with a framework-agnostic library that treats the entire system as the unit of analysis. Through a systematic system-level comparison across 3 benchmarks, 3 models, and 3 frameworks, we find that framework choice matters as much as model choice. MASEval allows researchers to explore all components of agentic systems, opening new avenues for principled system design, and practitioners to identify the best implementation for their use case. MASEval is available under the MIT licence https://github.com/parameterlab/MASEval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑