arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.07314cs.CLcs.AI
收藏

MEDIC:对LLM在临床应用中的安全性和实用性领先指标的综合评估

MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applications

M42

Praveenkumar Kanithi, Clément Christophe, Marco AF Pimentel, Tathagata Raha, Prateek Munjal, Nada Saadi, Hamza A Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, Shadab Khan

AI总结:

MEDIC通过综合评估框架揭示LLM在临床应用中的安全性和实用性差异,强调需采用组合方法以应对多维度性能权衡。

摘要:

尽管大型语言模型(LLMs)在标准化医学资格考试中表现出超人类的性能,但这些静态基准测试已变得饱和,并越来越与临床工作流程的功能需求脱节。为了弥合理论能力与已验证实用性之间的差距,我们引入MEDIC,一种综合评估框架,建立了跨各种临床维度的领先指标。除了标准的问题回答外,我们使用确定性执行协议和一种新的交叉检验框架(CEF)来评估操作能力,该框架在不依赖参考文本的情况下量化信息完整性和幻觉率。我们在异质任务套件上的评估揭示了关键的性能权衡:我们发现存在显著的知识执行差距,即在静态检索中的熟练程度并不能预测在诸如临床计算或SQL生成等操作任务中的成功。此外,我们观察到被动安全(拒绝)与主动安全(错误检测)之间的分歧,揭示了经过高拒绝率微调的模型往往无法可靠地审核临床文档的事实准确性。这些发现表明,没有任何单一架构在所有维度上都占主导地位,突显了在临床模型部署中采用组合方法的必要性。作为这项调查的一部分,我们发布了一个公开的排行榜,供Hugging Face使用。\footnote{https://huggingface.co/spaces/m42-health/MEDIC-Benchmark}

英文摘要:

While Large Language Models (LLMs) achieve superhuman performance on standardized medical licensing exams, these static benchmarks have become saturated and increasingly disconnected from the functional requirements of clinical workflows. To bridge the gap between theoretical capability and verified utility, we introduce MEDIC, a comprehensive evaluation framework establishing leading indicators of clinical LLM competence across five dimensions. These upfront indicators reveal cross-benchmark capability gaps, such as the divergence between static knowledge retrieval and functional execution, that inform model selection before costly deployment-based evaluation. Beyond standard question-answering, we assess operational capabilities using deterministic execution protocols and a novel Cross-Examination Framework (CEF), which quantifies information fidelity and hallucination rates without reliance on reference texts. Our evaluation across a heterogeneous task suite exposes critical performance trade-offs: we identify a significant knowledge-execution gap, where proficiency in static retrieval does not predict success in operational tasks such as clinical calculation or SQL generation. Furthermore, we observe a divergence between passive safety (refusal) and active safety (error detection), revealing that models fine-tuned for high refusal rates often fail to reliably audit clinical documentation for factual accuracy. These findings demonstrate that no single architecture dominates across all dimensions, highlighting the necessity of a portfolio approach to clinical model deployment. We accompany this work with a publicly available MEDIC leaderboard at https://hf.co/spaces/m42-health/MEDIC-Benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑