arXiv 2610.223502026-10-16cs.AIcs.CLcs.SE

FDE-Gym：AI交付的最后一公里

FDE-Gym: The Last Mile of AI Delivery

DeepLethe；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Beihang University（北京航空航天大学）；Xiamen University（厦门大学）；Nanjing University（南京大学）；Beijing Jiaotong University（北京交通大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Tianjin University（天津大学）；Shanghai University of Finance and Economics（上海财经大学）；Harvard University（哈佛大学）；Stanford University（斯坦福大学）；University College London（伦敦大学学院）；Cornell University（康奈尔大学）；Huawei（华为）；Atlassian；Meta；Microsoft Research Asia（微软亚洲研究院）；UWits Technology (Beijing) Co., Ltd.（优维思科技（北京）有限公司）；Bopai Management Consulting (Shenzhen) Co., Ltd.（博湃管理咨询（深圳）有限公司）；Exploration and Development Research Institute, PetroChina Southwest Oil & Gasfield Company（中国石油西南油气田公司勘探开发研究院）；Kunlun Digital Techn（昆仑数字技术）

Dongxu Yang, Hongda Yan, Renxin Wang, Baoli Zhang, Haojia Zuo, Ying Zhao, Youmin Chen, Xiaojian Liao, Congming Gao, Tianyang Jiang, Yixiao Chen, Qing Wang, Zhig… 展开作者

Dongxu Yang, Hongda Yan, Renxin Wang, Baoli Zhang, Haojia Zuo, Ying Zhao, Youmin Chen, Xiaojian Liao, Congming Gao, Tianyang Jiang, Yixiao Chen, Qing Wang, Zhiguo Yu, Xiaqing Li, Yuhao Zhang, Jiawen Sun, Renzhe Xu, De Mi, Tiancheng Wu, Xianya Fu, Xuanlu Zhao, Bo Feng, Meixuan Ren, Guanming Lin, Ziheng Zhou, Xinhua Wang

备注：: 14 pages, 8 figures, 4 tables. Website: https://fde-gym.com/paper ; Code: https://github.com/deeplethe/fde-gym ; Data: https://huggingface.co/datasets/DeepLethe/FDE-Gym

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

FDE-Gym是针对现实客户交付场景的智能体基准，含95项任务，评估显示现有模型在发现未明确约束等环节存在瓶颈，与解决明确任务的性能差距显著。

摘要：

现有的智能体基准通常针对由问题报告、规范或任务工单定义的明确任务来评估智能体。而现实世界中的客户交付场景有所不同：现场部署工程师（FDE）必须澄清模糊的需求、在实时系统中工作，并发现未明确说明的约束条件。我们推出FDE-Gym，这是一个包含95项端到端交付任务的基准，这些任务来自18个行业的20个客户场景，每项任务均由行业专家从现实世界的交付项目中改编而来。在具有脚本化利益相关者和可修改客户系统的模拟客户环境中，智能体与利益相关者交互并在类生产系统上操作，以交付可行的解决方案。FDE-Gym通过交付对客户自身成功指标的提升程度来评分，评分归一化范围为0（系统未发生变化）到1（参考解决方案），同时对利益相关者交互（消息、试运行和等待时间）进行惩罚。一个在线平台允许人类在相同的评分标准下练习FDE技能。我们评估了来自7家提供商的13种模型配置。在必须发现未明确约束的现实场景中，最佳配置的平均FDE-Gym评分为0.56，且仅解决了95项任务中的14项（评分至少为0.8且无事件）。发现未明确约束是主要瓶颈：当模型必须在无明确指导的情况下发现这些约束时，性能会急剧下降。然而，仅信息发现并不能解释这一差距：即使在最佳配置从利益相关者处获取所有关键事实的23项任务中，其归一化业务提升达到0.8或以上的也仅有10项。这些结果揭示了解决明确任务与为现实客户交付有效AI系统之间存在巨大差距。

英文摘要：

Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions (messages, trial runs, and waiting time). An online platform lets humans practice FDE skills under the same grading criteria. We evaluate 13 model configurations from seven providers. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks (a score of at least 0.8 with no incident). Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. Yet information discovery alone does not account for the gap: even on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。