CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?
Johns Hopkins Medicine(约翰霍普金斯医学中心);Wellstar Health System(Wellstar健康系统);Stanford University(斯坦福大学);CMU(卡内基梅隆大学);UCSD(加州大学圣地亚哥分校);Yale School of Medicine(耶鲁医学院);Salesforce AI Research(Salesforce人工智能研究);University of Washington(华盛顿大学);Northeastern University(东北大学);Brown University(布朗大学);Boston College(波士顿学院);Stony Brook University(史泰森布里克大学);University of Oxford(牛津大学);Arizona State University(亚利桑那州立大学);University of Southern California(南加州大学);Emory University(埃默里大学);MBZUAI;Recursive Superintelligence(递归超级智能);University of Illinois at Chicago(伊利诺伊大学芝加哥分校)
Haolin Chen, Deon Metelski, Leon Qi, Tao Xia, Joonyul Lee, Steve Brown, Kevin Riley, Frank Wang, T. Y. Alvin Liu, Hank Capps MD, Zeyu Tang, Xiangchen Song, Ling… 展开作者
Haolin Chen, Deon Metelski, Leon Qi, Tao Xia, Joonyul Lee, Steve Brown, Kevin Riley, Frank Wang, T. Y. Alvin Liu, Hank Capps MD, Zeyu Tang, Xiangchen Song, Lingjing Kong, Fan Feng, Tianyi Zeng, Zhiwei Liu, Zixian Ma, Hang Jiang, Fangli Geng, Yuan Yuan, Chenyu You, Qingsong Wen, Hua Wei, Yanjie Fu, Yue Zhao, Carl Yang, Biwei Huang, Kun Zhang, Caiming Xiong, Sanmi Koyejo, Eric P. Xing, Philip S. Yu, Weiran Yao
End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks: policy density, decisions must be grounded in a large library of medical, insurance, and operational rules; Multi-role composition: a single task requires the agent to play multiple roles with handoffs; and multilateral interaction: intermediate workflow steps are multi-turn dialogs, such as peer-to-peer review and patient outreach. We introduce $χ$-Bench, a benchmark of long-horizon healthcare workflows across three domains: provider prior authorization, payer utilization management, and care management. Each task hands the agent a clinical case in a high-fidelity simulator of 20 healthcare apps exposed via 87 MCP tools, which it must drive to a terminal status through tool calls and writing the role's artifacts, guided by a 1,290+ document managed-care operations handbook skill. Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass^3, and executing all tasks in a single session slumps the performance to 3.8%. These results raise the hypothesis that similar gaps are likely to surface in other policy-dense, role-composed, irreversible enterprise domains.
AIR-Bench: Automated Heterogeneous Information Retrieval Benchmark
University of Science and Technology of China(中国科学技术大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Jina AI(极光科技);Beijing University of Posts and Telecommunications(北京邮电大学);Shenzhen University(深圳大学);State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);Hong Kong Polytechnic University(香港理工大学)