EHR-Complex：面向复杂临床推理的医疗智能体基准测试
EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning
AI总结：
提出EHR-Complex基准，基于MIMIC-IV构建约5.2万项交互式临床数据库推理任务，评估智能体在复杂SQL和代码执行中的表现，揭示主要失败模式。
摘要：
临床智能体有望普及电子健康记录（EHR）的访问，但现有基准未能反映实际EHR分析的复杂性，例如通常通过静态SQL生成在理想化、干净的EHR上操作，而非交互式执行。在这项工作中，我们引入了EHR-Complex，一个专为交互式临床数据库推理设计的大规模基准。基于大型MIMIC-IV数据集（36.5万患者、31张表、5亿+记录），EHR-Complex包含约5.2万项任务，涵盖六种临床意图，支持患者级和人群级查询，每项任务要求智能体通过执行SQL查询或Python代码与沙盒环境交互。值得注意的是，EHR-Complex考虑了现实世界SQL任务的复杂性，包括纵向多表聚合和组合推理，平均每个查询包含31.93个SQL结构组件。在EHR-Complex上的评估结果揭示了这些EHR推理场景的临床难度，表现最佳的模型仅达到62.3%的精确匹配准确率。几乎所有评估模型在k=4时的Pass^k一致性低于50%，暴露出广泛的随机脆弱性。对代表性LLM的3800多条失败轨迹进行细粒度分析，揭示了三种主要失败模式：SQL逻辑错误、医学术语查找失败和语义误解。EHR-Complex为临床智能体提供了严格的测试平台，并突显了大规模EHR分析中稳健推理的现有差距。
英文摘要：
Clinical agents promise to democratize access to electronic health records (EHRs), yet existing benchmarks fail to reflect the complexity of practical EHR analysis, e.g., often operating on idealized, clean EHRs via static SQL generation rather than interactive execution. In this work, we introduce EHR-Complex, a large-scale benchmark designed for interactive clinical database reasoning. Built on the large MIMIC-IV substrate (365K patients, 31 tables, 500M+ records), EHR-Complex comprises about 52K tasks spanning six clinical intents, supporting both patient-level and population-level queries, where each task requires an agent to interact with a sandboxed environment by executing SQL queries or Python code. Notably, EHR-Complex considers the real-world SQL task complexity for longitudinal multi-table aggregation and compositional reasoning, resulting in 31.93 SQL structural components per query on average. Evaluation results on EHR-Complex reveal the clinical difficulty of these EHR reasoning scenarios, with the top-performing model achieving only 62.3% exact-match accuracy. Pass^k consistency drops below 50% for nearly all evaluated models at k=4, exposing broad stochastic fragility. A fine-grained analysis of more than 3,800 failed trajectories for representative LLMs reveals three dominant failure modes: SQL logic errors, medical-code lookup failures, and semantic misunderstandings. EHR-Complex provides a rigorous testbed for clinical agents and highlights remaining gaps in robust reasoning for large-scale EHR analysis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
