arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.03596cs.AIcs.CLcs.DBcs.LG
收藏

Workspace-Bench 1.0:基于大规模文件依赖的AI代理工作空间基准测试

Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies

Zirui Tang, Xuanhe Zhou, Yumou Liu, Linchun Li, Yukai Wu, Weizheng Wang, Hongzhang Huang, Wei Zhou, Jun Zhou, Jiachen Song, Shaoli Yu, Jinqi Wang, Zihang Zhou, … 展开作者

Zirui Tang, Xuanhe Zhou, Yumou Liu, Linchun Li, Yukai Wu, Weizheng Wang, Hongzhang Huang, Wei Zhou, Jun Zhou, Jiachen Song, Shaoli Yu, Jinqi Wang, Zihang Zhou, Hongyi Zhou, Yuting Lv, Jinyang Li, Jiashuo Liu, Ruoyu Chen, Chunwei Liu, GuoLiang Li, Jihua Kang, Fan Wu

AI总结:

本文提出Workspace-Bench 1.0,通过构建包含5个工人配置、74种文件类型和20,476个文件的现实工作空间,评估AI代理在处理大规模文件依赖任务中的能力,发现当前代理在工作空间学习中表现远低于人类水平。

摘要:

Workspace学习要求AI代理能够识别、推理解析、利用并更新工人工作空间中异构文件之间的显式和隐式依赖关系,从而有效完成常规和高级任务。尽管其重要性,现有相关基准大多评估代理在预定义或合成文件上的性能,缺乏对工作空间层面的评估。为此,我们引入Workspace-Bench,一个用于评估涉及大规模文件依赖的AI代理工作空间学习的基准。我们构建了包含5个工人配置、74种文件类型、20,476个文件(最多20GB)的真实工作空间,并精心挑选了388个任务,每个任务都有其自身的文件依赖图,评估跨7,399个总评价标准,要求跨文件检索、上下文推理和适应性决策。我们进一步提供Workspace-Bench-Lite,一个100任务子集,保留基准分布的同时将评估成本降低约70%。我们评估了4种流行的代理框架和7种基础模型。实验结果表明,当前代理在工作空间学习中仍远未可靠,最佳表现仅达到约60%,显著低于人类结果80.7%,各代理的平均性能仅为43.3%。

英文摘要:

Workspace learning requires AI agents to identify, reason over, exploit, and update explicit and implicit dependencies among heterogeneous files in a worker's workspace, enabling them to complete both routine and advanced tasks effectively. Despite its importance, existing relevant benchmarks largely evaluate agents on pre-specified or synthesized files with limited real-world dependencies, leaving workspace-level evaluation underexplored. To this end, we introduce Workspace-Bench, a benchmark for evaluating AI agents on Workspace Learning involving Large-Scale File Dependencies. We construct realistic workspaces with 5 worker profiles, 74 file types, 20,476 files (up to 20GB) and curate 388 tasks, each with its own file dependency graph, evaluated across 7,399 total rubrics that require cross-file retrieval, contextual reasoning, and adaptive decision-making. We further provide Workspace-Bench-Lite, a 100-task subset that preserves the benchmark distribution while reducing evaluation costs by about 70%. We evaluate 4 popular agent harnesses and 7 foundation models. Experimental results show that current agents remain far from reliable workspace learning, where the best reaches only about 60%, substantially below the human result of 80.7%, and the average performance across agents is only 43.3%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑