EvoCowork:面向真实世界办公任务的基准测试集
EvoCowork: A Benchmark for Real-World Office Tasks
AI总结:
本研究推出含240项任务的EvoCowork基准测试集,评估模型权重固定时办公场景下的框架级递归自改进,发现反馈文件的性能提升未必迁移至跨办公协同工作流。
摘要:
办公工作流是对自改进智能体的严苛测试平台,因为其交付成果必须正确、可原生编辑且相互一致。核心问题在于,反馈任务上的性能提升是否反映了可复用的流程,而非针对特定任务的调整。我们推出EvoCowork,这是一个包含240项任务的基准测试集,用于在模型权重固定的情况下,评估四种办公模态下的框架级递归自改进能力。该基准测试集涵盖演示文稿、文档、电子表格及跨办公工作流,会根据内容、原生结构、公式、跨文件一致性及必要时的视觉质量对提交的成果进行评分。其协议将反馈、修订选择与最终测试分离:方法先从单模态任务中学习,再在共享初始框架和明确预算下,针对未见过的单模态任务及跨办公工作流进行评估。在对两个模型和四个框架的静态评估中,最佳配置的总分为42.25,跨办公任务得分为30.62。从搭载GLM 5.3的相同MSA框架出发,ACE和AHE的域内得分分别提升4.90和2.64分,但跨办公得分仅分别变化+0.03和-1.72。因此,提供反馈的文件类型上的提升未必能迁移到协同工作流。EvoCowork明确了这一区别,同时记录框架演化的能力、迁移性及成本。代码和任务包:此https URL。
英文摘要:
Office workflows are a demanding testbed for self-improving agents because their deliverables must be correct, natively editable, and mutually consistent. A central question is whether gains on feedback tasks reflect reusable procedures rather than task-specific fixes. We introduce EvoCowork, a 240-task benchmark for evaluating harness-level recursive self-improvement across four office modalities, with model weights held fixed. The benchmark covers presentations, documents, spreadsheets, and cross-office workflows, and scores submitted artifacts for content, native structure, formulas, cross-file consistency, and visual quality where needed. Its protocol separates feedback, revision selection, and final testing: methods learn from single-modality tasks, then are evaluated on unseen single-modality tasks and cross-office workflows under a shared initial harness and explicit budgets. In static evaluations of two models and four harnesses, the best configuration scores 42.25 overall and 30.62 on cross-office tasks. Starting from the same MSA harness with GLM 5.3, ACE and AHE improve in-domain scores by 4.90 and 2.64 points, respectively, but change cross-office scores by only +0.03 and -1.72. Thus, gains on the file types that provide feedback do not necessarily transfer to coordinated workflows. EvoCowork makes this distinction explicit while recording the capability, transfer, and cost of harness evolution. Code and task packages: https://github.com/DeepThinkingZhouLiu/cowork-evolution-benchmark.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。