HealthAgentBench：面向挑战性前沿AI代理的统一真实医疗环境基准套件
HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents
AI总结：
提出包含54个医疗任务的基准套件HealthAgentBench，评估AI代理在真实临床工作流中的端到端多步推理能力，最强代理Codex GPT-5.5仅达42%成功率。
摘要：
随着AI代理在复杂、长期推理方面变得日益强大，严格且全面的评估对于衡量在现实世界医疗应用中的进展至关重要。我们推出HealthAgentBench，这是一个包含54个代理医疗任务的套件，涵盖7个类别，每个类别都有其独特的环境。该基准套件跨越患者旅程中的多样化工作流程和广泛模态。每个任务旨在复制端到端的临床工作流：给定最少的指令，代理必须探索原始医疗数据，在复杂环境中操作，并执行超越简单提示的多步解决方案。最终报告任务成功率，为每个代理提供HealthAgentBench整体性能的单一、可解释指标。在前沿代理上评估HealthAgentBench，我们发现总体任务成功率仍然很低，突显了该套件的难度。最强且最具成本效益的代理Codex GPT-5.5仅达到约42%的成功率。除了总体性能，HealthAgentBench还揭示了跨任务类别的细微优势和劣势。前沿代理在自动开发基于EHR数据的研究建模流程方面显示出潜力，但医学成像仍然特别具有挑战性，尤其是对Claude Code模型，而Codex GPT-5.5显示出新兴能力。结合大搜索空间和组合推理要求的任务对所有当前代理仍然困难。这些结果表明，HealthAgentBench提供了一个具有挑战性和现实性的基准，未来仍有很大的改进空间。我们在以下网址发布基准：https://this https URL。
英文摘要：
As AI agents become increasingly capable of complex, long-horizon reasoning, rigorous and holistic evaluation is essential for measuring progress toward real-world healthcare applications. We introduce HealthAgentBench, a suite of 54 agentic healthcare tasks across 7 categories each with its unique environment. The benchmark suite spans diverse workflows throughout the patient journey and a broad range of modalities. Each task is designed to replicate an end-to-end clinical workflow: given minimal instructions, an agent must explore raw healthcare data, operate within a complex environment, and execute multi-step solutions that go beyond naive prompting. A final task success rate is reported to provide a single, interpretable metric for HealthAgentBench overall performance for each agent. Evaluating frontier agents on HealthAgentBench, we find that overall task success rate remains low, underscoring the difficulty of the suite. The strongest and the most cost effective agent, Codex GPT-5.5, achieves only approximately 42% success rate. Beyond aggregate performance, HealthAgentBench reveals nuanced strengths and weaknesses across task categories. Frontier agents show promise in automatically developing research modeling pipelines over EHR data, but medical imaging remains especially challenging, particularly for Claude Code models, while Codex GPT-5.5 shows emerging capability. Tasks that combine large search spaces with compositional reasoning requirements remain difficult for all current agents. Together, these results suggest that HealthAgentBench provides a challenging and realistic benchmark with substantial room for future progress. We release our benchmark at https://github.com/microsoft/HealthAgentBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
