arXiv 2602.112242026-02cs.SEcs.CL

Agent-Diff：通过代码执行基于状态差的评估来基准测试LLM代理在企业API任务上的表现

Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation

Minerva University（Minerva大学）

Hubert M. Pysklo, Artem Zhuravel, Patrick D. Watson

备注：: Pre-Print. Under review for KDD 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Agent-Diff框架，通过代码执行和基于状态差的评估，评估LLM代理在企业API任务中的性能，提供了九个LLM在224个任务上的基准测试，并通过消融实验评估框架的鲁棒性。

摘要：

我们提出了Agent-Diff，一种新颖的基准测试框架，用于通过代码执行评估代理式大语言模型（LLM）在真实世界生产力软件API任务上的表现。代理式LLM的性能因模型差异、外部工具访问、提示结构和代理框架而异。基准测试必须在受控的沙盒方法和更生态有效的实际服务方法之间做出根本性权衡。Agent-Diff试图通过提供对真实API接口的访问同时沙盒化调用、处理和评估的环境来结合这两种方法的有利特征。这种方法依赖于两个关键创新。第一个是新颖的状态差合同，它将过程与结果分开——而不是模糊轨迹或参数匹配，我们定义任务成功为是否实现了预期的环境状态变化。第二个是基于容器化企业API副本的新型沙盒，允许所有模型通过代码执行交互相同的API接口。这使能够针对共同的状态差合同进行受控评估，同时保留现实世界API交互的结构。使用Agent-Diff框架，我们为九个LLM提供了224个任务的基准测试，利用企业软件工作流。此外，我们通过消融实验评估框架的鲁棒性，以评估对API文档访问的贡献。代码和数据：https://github.com/agent-diff-bench/agent-diff.

英文摘要：

We present Agent-Diff, a novel benchmarking framework for evaluating agentic Large Language Models (LLMs) on real-world productivity software API tasks via code execution. Agentic LLM performance varies due to differences in models, external tool access, prompt structures, and agentic frameworks. Benchmarks must make fundamental trade-offs between a sandboxed approach that controls for variation in software environments and more ecologically valid approaches employing real services. Agent-Diff attempts to capture the desirable features of both of these approaches by including access to the real API interfaces for software services while sandboxing the environment in which calls are made, processed, and evaluated. This approach relies on two key innovations. The first is a novel state-diff contract, which separates process from outcome - rather than fuzzy trace or parameter matching, we define task success as whether the expected change in environment state was achieved. The second is a novel sandbox built on containerized replicas of enterprise APIs, allowing all models to interact with the same service interfaces through code execution. This enables controlled evaluation against a common set of state-diff contracts while preserving the structure of real-world API interaction. Using the Agent-Diff framework, we provide benchmarks for nine LLMs across 224 tasks utilizing enterprise software workflows. In addition, we evaluate the robustness of the framework with ablation experiments to assess the contribution of access to API documentation on benchmark performance. Code and data: https://github.com/agent-diff-bench/agent-diff.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文