arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.22953cs.AI
收藏

通用智能体评估

General Agent Evaluation

IBM Research(IBM研究院);MIT(麻省理工学院)
Elron Bandel, Asaf Yehudai, Lilach Eden, Yehoshua Sagron, Yotam Perlitz, Elad Venezian, Natalia Razinkov, Natan Ergas, Shlomit Shachor Ifergan, Segev Shlomov, M… 展开作者

Elron Bandel, Asaf Yehudai, Lilach Eden, Yehoshua Sagron, Yotam Perlitz, Elad Venezian, Natalia Razinkov, Natan Ergas, Shlomit Shachor Ifergan, Segev Shlomov, Michal Jacovi, Leshem Choshen, Liat Ein-Dor, Yoav Katz, Michal Shmueli-Scheuer

AI总结:

本文系统评估了通用智能体在不同协议和环境下的性能,提出统一协议和评估框架,揭示了智能体架构和基础模型对性能的影响,发现通用智能体无需定制即可适应多种领域。

摘要:

通用智能体能够在不熟悉环境中执行任务而无需领域特定的手动定制。然而,尚未有系统研究评估智能体架构在异质协议和多样环境下的性能。本文首次系统研究了工具调用、MCP、代码生成和CLI智能体在同一基准上的表现。通过统一协议和评估框架,发现通用智能体无需定制即可适应所有测试领域;智能体架构选择对结果影响显著,但基础模型选择主导整体性能;在四个基准测试中,顶级通用智能体与高度定制的领域特定智能体表现相似;开放权重模型表现出“一般性陷阱”,而前沿闭源模型则无此问题;行为失败分析揭示了架构特有的错误特征,聚合评分无法区分。代码、框架、排行榜和追踪信息见https://www.exgentic.ai.

英文摘要:

General-purpose agents perform tasks in unfamiliar environments without domain-specific manual customization. Yet no study has systematically measured how agent architecture shapes performance across heterogeneous protocols and diverse unfamiliar environments. This is the first systematic study, comparing tool-calling, MCP, code-generation, and CLI agents on the same benchmarks with the same models. Two gaps blocked such a study: existing harnesses require per-benchmark wiring or fixed protocol classes (web for BrowserGym, CLI for Harbor), and benchmarks themselves expect human-authored prompts, context, and integration glue. To enable this study, we contribute (1) a unifying protocol that bridges existing benchmark and agent protocols; (2) an evaluation harness that surfaces any benchmark to any general-purpose agent and backbone model; and (3) the first Open General Agent Leaderboard of agent configurations, a full factorial over 5 agent architectures x 5 backbone LLMs (three closed-source, two open-weight) x 6 benchmarks spanning software engineering, customer service, deep research, and personal assistance. We find that (i) general agents adapt to every tested domain without per-domain customization; (ii) agent architecture choice swings results by up to 12pp within a single model, yet backbone model choice dominates overall performance; (iii) on 4 of 6 tested benchmarks, top general agents are indistinguishable from the leading heavily-customized domain-specific agents; (iv) open-weight models tested exhibit "generality sinks" absent from frontier closed-source models: they consistently collapse on specific agent architectures or benchmarks; (v) a behavioral failure analysis reveals architecture-distinctive error signatures that aggregate scoring cannot discriminate. Code, harness, leaderboard, and traces are at https://www.exgentic.ai.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑