arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.16679cs.CLcs.AI
收藏

CHI-Bench: 能否让AI代理自动化端到端、长周期、政策丰富的医疗工作流程?

CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?

Johns Hopkins Medicine(约翰霍普金斯医学中心);Wellstar Health System(Wellstar健康系统);Stanford University(斯坦福大学);CMU(卡内基梅隆大学);UCSD(加州大学圣地亚哥分校);Yale School of Medicine(耶鲁医学院);Salesforce AI Research(Salesforce人工智能研究);University of Washington(华盛顿大学);Northeastern University(东北大学);Brown University(布朗大学);Boston College(波士顿学院);Stony Brook University(史泰森布里克大学);University of Oxford(牛津大学);Arizona State University(亚利桑那州立大学);University of Southern California(南加州大学);Emory University(埃默里大学);MBZUAI;Recursive Superintelligence(递归超级智能);University of Illinois at Chicago(伊利诺伊大学芝加哥分校)
Haolin Chen, Deon Metelski, Leon Qi, Tao Xia, Joonyul Lee, Steve Brown, Kevin Riley, Frank Wang, T. Y. Alvin Liu, Hank Capps MD, Zeyu Tang, Xiangchen Song, Ling… 展开作者

Haolin Chen, Deon Metelski, Leon Qi, Tao Xia, Joonyul Lee, Steve Brown, Kevin Riley, Frank Wang, T. Y. Alvin Liu, Hank Capps MD, Zeyu Tang, Xiangchen Song, Lingjing Kong, Fan Feng, Tianyi Zeng, Zhiwei Liu, Zixian Ma, Hang Jiang, Fangli Geng, Yuan Yuan, Chenyu You, Qingsong Wen, Hua Wei, Yanjie Fu, Yue Zhao, Carl Yang, Biwei Huang, Kun Zhang, Caiming Xiong, Sanmi Koyejo, Eric P. Xing, Philip S. Yu, Weiran Yao

AI总结:

本文提出CHI-Bench基准,旨在评估AI代理在医疗工作流程中端到端、长周期和政策丰富任务中的自动化能力,揭示当前基准测试中政策密度、多角色协作和多方交互等能力的不足。

摘要:

现实医疗操作的端到端自动化要求具备当前基准测试中较少体现的三种能力:政策密度,即决策必须基于大量医疗、保险和运营规则;多角色组成,即单个任务需要代理扮演多个角色并进行交接;以及多方交互,即中间工作流程步骤是多轮对话,例如同行评审和患者接触。我们介绍了CHI-Bench,一个涵盖三个领域的长周期医疗工作流程基准:提供者预先授权、支付方使用管理以及护理管理。每个任务都会将代理置于一个高保真模拟器中,该模拟器暴露了20个医疗应用程序,通过87个MCP工具。代理必须通过工具调用和编写角色的文档来驱动任务完成,受1,290多份文档管理护理操作手册技能的指导。在30种代理配置下,最佳代理仅能解决28.0%的任务,没有代理在严格通过标准下达到20%以上,且单次会话执行所有任务会将性能降至3.8%。这些结果提出了假设,即在其他政策密集、角色组成和不可逆的企业领域中,类似的差距可能会出现。

英文摘要:

End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks: policy density, decisions must be grounded in a large library of medical, insurance, and operational rules; Multi-role composition: a single task requires the agent to play multiple roles with handoffs; and multilateral interaction: intermediate workflow steps are multi-turn dialogs, such as peer-to-peer review and patient outreach. We introduce $χ$-Bench, a benchmark of long-horizon healthcare workflows across three domains: provider prior authorization, payer utilization management, and care management. Each task hands the agent a clinical case in a high-fidelity simulator of 20 healthcare apps exposed via 87 MCP tools, which it must drive to a terminal status through tool calls and writing the role's artifacts, guided by a 1,290+ document managed-care operations handbook skill. Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass^3, and executing all tasks in a single session slumps the performance to 3.8%. These results raise the hypothesis that similar gaps are likely to surface in other policy-dense, role-composed, irreversible enterprise domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑