超越排行榜：重新思考大型语言模型的医疗基准
Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models
AI总结：
本文提出MedCheck框架，用于评估医疗领域大型语言模型的基准，揭示现有基准在临床相关性、数据完整性及安全性方面的系统性问题。
摘要：
大型语言模型（LLMs）在医疗领域展现出巨大潜力，促使众多基准用于评估其能力。然而，这些基准的可靠性仍存疑，往往缺乏临床真实性、稳健的数据管理和以安全为导向的评估指标。为解决这些不足，我们引入MedCheck，首个生命周期导向的评估框架，专门设计用于医疗基准。我们的框架将基准的开发分解为五个连续阶段，从设计到治理，并提供46项医学定制的检查清单。利用MedCheck，我们对53个医疗LLM基准进行了深入实证评估。我们的分析揭示了广泛存在的系统性问题，包括与临床实践的深刻脱节、由于未解决的污染风险导致的数据完整性危机，以及对安全关键评估维度如模型鲁棒性和不确定性意识的系统性忽视。基于这些发现，MedCheck既是对现有基准的诊断工具，也是促进更标准化、可靠和透明评估AI在医疗领域应用的行动指南。
英文摘要：
Large language models (LLMs) show significant potential in healthcare, prompting numerous benchmarks to evaluate their capabilities. However, concerns persist regarding the reliability of these benchmarks, which often lack clinical fidelity, robust data management, and safety-oriented evaluation metrics. To address these shortcomings, we introduce MedCheck, the first lifecycle-oriented assessment framework specifically designed for medical benchmarks. Our framework deconstructs a benchmark's development into five continuous stages, from design to governance, and provides a comprehensive checklist of 46 medically-tailored criteria. Using MedCheck, we conducted an in-depth empirical evaluation of 53 medical LLM benchmarks. Our analysis uncovers widespread, systemic issues, including a profound disconnect from clinical practice, a crisis of data integrity due to unmitigated contamination risks, and a systematic neglect of safety-critical evaluation dimensions like model robustness and uncertainty awareness. Based on these findings, MedCheck serves as both a diagnostic tool for existing benchmarks and an actionable guideline to foster a more standardized, reliable, and transparent approach to evaluating AI in healthcare.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
