arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.15239cs.AI
收藏

ClinEdge:前沿多模态模型失效的真实患者边缘案例的专家分级基准

ClinEdge: An Expert-Graded Benchmark of Real Patient Edge Cases Where Frontier Multimodal Models Fail

FutrBridge

Anurag Sharma, Aditya Sharma, Sapna Baalwan

AI总结:

ClinEdge 构建含 32 个真实患者案例的基准,评估 7 个前沿多模态模型,发现其存在 18 种共性失效模式,邀请相关机构评估模型性能

摘要:

目前大多数医学领域的大语言模型基准构建自执业资格考试和理论问题,少数利用真实临床数据的基准也仅部分使用,且很少要求模型读取患者自身的影像数据。ClinEdge 填补了这一空白,包含 32 个来自活跃临床实践的去标识化真实患者案例,每个案例在 7 个前沿系统上接受评估,共完成 224 次专家分级评估。每个案例将主要影像与简短 vignette(临床 vignette 指临床场景描述)配对,包含已知会击败模式匹配推理的特征,并根据患者记录的临床进程评分。该基准套件注重深度而非数量,分级遵循两阶段专家流程:首先,一名高级骨科医生使用加权评分标准对全部 224 次响应进行分级,给出通过/未通过的判定;其次,由 19 名主要为医学院校高级专科医生组成的小组,独立审查每个案例及全部 224 次分级。所有 7 个生产系统均通过其公共接口进行评估,全部 224 次对话均以可重放链接形式保存。所有系统的表现均不佳,且失效模式具有共性:不同厂商遗漏相同的发现,且趋向相同的错误方案。从专家笔记中,我们梳理出 5 个类别中的 18 种反复出现的失效模式:模型在发现首个结果时便停止读取影像;给出通用方案并倾向于过度治疗;信任误导性病史而非正确的影像;不同系统及不同运行的答案存在差异,且随问题格式变化;错误答案表述流畅且未表现出任何不确定性。4 个案例已完整发布,剩余 28 个案例保留,我们邀请前沿 AI 实验室和医疗 AI 公司针对这些案例评估其模型(邮箱:research@futrbridge.com)

英文摘要:

Most benchmarks for large language models in medicine are built from licensing examinations and theoretical questions. The few that draw on real clinical data do so only in part, and rarely require a model to read a patient's own imaging. ClinEdge addresses this gap with 32 real, de-identified patient cases from active clinical practice, each evaluated across seven frontier systems for a total of 224 expert-graded evaluations. Each case pairs primary imaging with a short vignette, contains a feature known to defeat pattern-matched reasoning, and is scored against the patient's documented clinical course. The suite is built for depth rather than volume. Grading followed a two-stage expert process. First, a senior orthopaedic surgeon graded all 224 responses on a weighted rubric and a pass-or-fail verdict. Second, a panel of nineteen physicians, largely senior specialists at teaching medical colleges, independently reviewed every case and every one of the 224 grades. All seven production systems were evaluated through their public interfaces, and all 224 conversations are preserved as replayable links. Performance was poor across all systems, and the failures were shared: different vendors missed the same findings and converged on the same wrong plans. From the experts' notes we catalogue 18 recurring failure modes in five families. The models stop reading an image at the first finding. They give generic plans and lean toward over-treatment. They trust a misleading history over correct imaging. Their answers vary across systems and runs, and change with question format. Failing answers were delivered fluently and without any expressed uncertainty. Four cases are released in full; the remaining 28 are held out, and we invite frontier AI labs and healthcare AI companies to evaluate their models against them (research@futrbridge.com)

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑