arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.14539cs.AIcs.MA
收藏

FDE-Bench:评估来自欠规范真实世界业务请求的端到端交付

FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests

FM Agent Team, Baidu AI Cloud(百度智能云FM智能体团队);Beijing University of Posts and Telecommunications(北京邮电大学);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学);Peking University(北京大学);University of the Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Huaiming Li, Can Huang, Chufan Wu, Danyu Liu, Xiaomin Yuan, Qinze Li, Yui Lo, Wenhui Bai, Qianlong Wang, Zengle Ge, Xiyu Yan, Yuheng Cheng, Jinhui Ren, Guokai C… 展开作者

Huaiming Li, Can Huang, Chufan Wu, Danyu Liu, Xiaomin Yuan, Qinze Li, Yui Lo, Wenhui Bai, Qianlong Wang, Zengle Ge, Xiyu Yan, Yuheng Cheng, Jinhui Ren, Guokai Chen, Jiaqun Liu, Haoran Li, Qianhui Liu, Annan Li, Jianmin Wu, Dawei Yin, Dou Shen

AI总结:

FDE-Bench是首个基于真实客户项目的基准,评估智能体在欠规范业务任务中交互式需求澄清与迭代优化的端到端交付能力,实验显示现有智能体表现远低于FDE参考水平。

摘要:

真实世界的业务请求常常未明确关键需求,因为一线从业者难以清晰表述他们从经验中掌握的规则。前置部署工程师(FDEs)必须通过对话引出这些需求,并迭代优化解决方案,直至满足客户验收标准。现有的优化基准通常从明确的目标出发,而澄清基准往往在现有任务中引入信息缺口。我们推出FDE-Bench,这是首个基于真实客户项目构建的基准,用于评估智能体在需要交互式需求澄清和迭代优化的端到端交付任务中的表现。其包含49个案例,涵盖组合优化和机器学习领域。澄清目标源自一线从业者与FDEs的讨论。智能体从模拟客户处引出缺失的需求,而针对具体案例的评估者则以客户认可的FDE解决方案作为质量参考,依据验收标准评估最终成果。在18种智能体配置中,最高平均交付分数仅为0.5510,而FDE参考值为1.0;最高交付成功率(DSR,定义为达到该参考值的计划运行比例)仅为6.12%。这些结果表明,在欠规范的真实世界业务任务中,被评估的智能体仍无法实现可靠的FDE级端到端交付。该基准项目页面可访问此https URL。

英文摘要:

Real-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experience. Forward-deployed engineers (FDEs) must elicit these requirements through dialogue and iteratively optimize solutions until they meet customer acceptance criteria. Existing optimization benchmarks typically start from explicit objectives, while clarification benchmarks often introduce information gaps into existing tasks. We introduce FDE-Bench, the first benchmark built from real customer projects to evaluate agents on end-to-end delivery tasks that require interactive requirement clarification and iterative optimization. Its 49 cases span combinatorial optimization and machine learning. Clarification targets are distilled from discussions between frontline practitioners and FDEs. Agents elicit missing requirements from a simulated customer, and case-specific evaluators assess final artifacts against acceptance criteria using customer-accepted FDE solutions as quality references. Across 18 agent configurations, the highest mean delivery score is only 0.5510, compared with an FDE reference of 1.0. The highest delivery success rate (DSR), defined as the fraction of planned runs reaching this reference, is only 6.12%. These results show that reliable FDE-level end-to-end delivery remains out of reach for the evaluated agents on underspecified real-world business tasks. The benchmark project page is available at https://baidubce.github.io/FM-Agent/FDE-Bench/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑