FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests
FM Agent Team, Baidu AI Cloud(百度智能云FM智能体团队);Beijing University of Posts and Telecommunications(北京邮电大学);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学);Peking University(北京大学);University of the Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Real-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experience. Forward-deployed engineers (FDEs) must elicit these requirements through dialogue and iteratively optimize solutions until they meet customer acceptance criteria. Existing optimization benchmarks typically start from explicit objectives, while clarification benchmarks often introduce information gaps into existing tasks. We introduce FDE-Bench, the first benchmark built from real customer projects to evaluate agents on end-to-end delivery tasks that require interactive requirement clarification and iterative optimization. Its 49 cases span combinatorial optimization and machine learning. Clarification targets are distilled from discussions between frontline practitioners and FDEs. Agents elicit missing requirements from a simulated customer, and case-specific evaluators assess final artifacts against acceptance criteria using customer-accepted FDE solutions as quality references. Across 18 agent configurations, the highest mean delivery score is only 0.5510, compared with an FDE reference of 1.0. The highest delivery success rate (DSR), defined as the fraction of planned runs reaching this reference, is only 6.12%. These results show that reliable FDE-level end-to-end delivery remains out of reach for the evaluated agents on underspecified real-world business tasks. The benchmark project page is available at https://baidubce.github.io/FM-Agent/FDE-Bench/.
DeepLethe;Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);Beihang University(北京航空航天大学);Xiamen University(厦门大学);Nanjing University(南京大学);Beijing Jiaotong University(北京交通大学);Beijing University of Posts and Telecommunications(北京邮电大学);Tianjin University(天津大学);Shanghai University of Finance and Economics(上海财经大学);Harvard University(哈佛大学);Stanford University(斯坦福大学);University College London(伦敦大学学院);Cornell University(康奈尔大学);Huawei(华为);Atlassian;Meta;Microsoft Research Asia(微软亚洲研究院);UWits Technology (Beijing) Co., Ltd.(优维思科技(北京)有限公司);Bopai Management Consulting (Shenzhen) Co., Ltd.(博湃管理咨询(深圳)有限公司);Exploration and Development Research Institute, PetroChina Southwest Oil & Gasfield Company(中国石油西南油气田公司勘探开发研究院);Kunlun Digital Techn(昆仑数字技术)
School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));King Abdullah University of Science and Technology(国王 Abdullah 科学与技术大学)
FDE-Bench: Evaluating LLM Agents for Deployment Environment Configuration
University of California, Berkeley(加州大学伯克利分校);Imperial College London(帝国理工学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
No More, No Less: Task Alignment in Terminal Agents
CISPA Helmholtz Center for Information Security(CISPA 涅槃中心信息安全研究所);ELLIS Institute Tübingen and MPI for Intelligent Systems(图宾根 ELLIS 研究所和智能系统 MPI);Tübingen AI Center(图宾根人工智能中心)
AI Evaluation Should Require Standardized Item-Level Data Releases
Johns Hopkins University(约翰霍普金斯大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Microsoft Research Asia(微软亚洲研究院);Stanford University(斯坦福大学);North Carolina State University(北卡罗来纳州立大学);Tsinghua University(清华大学)
MIRA: A Musical Intent Refinement Agent for Aligning Text-to-Music Generation with User Intent
Shandong University(山东大学);University of Science and Technology of China(中国科学技术大学);Central Conservatory of Music(中央音乐学院);Kunlun Tech Co. Ltd.(昆仑万维科技股份有限公司);Shanghai Jiao Tong University(上海交通大学)
$π$-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Fudan University(复旦大学);University of Science and Technology of China(中国科学技术大学);Peking University(北京大学);Nanjing University(南京大学);Zhejiang University(浙江大学);Tongji University(同济大学);Soochow University(苏州大学);The Chinese University of Hong Kong(香港中文大学)
SciRisk-Bench: A Risk-Dimension-Aware Benchmark for AI4Science Safety
Brain-inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China(脑启发认知智能实验室,自动化研究所,中国科学院,北京,中国);School of Future Technology, University of Chinese Academy of Sciences, China(未来技术学院,中国科学院大学,中国);School of Artificial Intelligence, University of Chinese Academy of Sciences, China(人工智能学院,中国科学院大学,中国);Zhongguancun Academy, China(中关村学院,中国);Beijing Key Laboratory of Safe AI and Superalignment(北京安全人工智能与超对齐重点实验室);Gaoling School of AI, Renmin University of China(甘露人工智能学院,中国人民大学);Beijing Institute of AI Safety and Governance (Beijing-AISI)(北京人工智能安全与治理研究院(北京-AISI));School of Humanities, University of Chinese Academy of Sciences, China(人文学院,中国科学院大学,中国)
Evaluating LLMs on Conversational Text-to-SQL under Chain Ambiguity and Intent Drift
The Hong Kong Polytechnic University(香港理工大学);City University of Macau(澳门城市大学);Jilin University(吉林大学);Beihang University(北京航空航天大学);Jinan University(暨南大学)