发表机构
Zhejiang University; Ant Healthcare, Ant Group（浙江大学; 蚂蚁集团蚂蚁医疗）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对动态医学诊断中证据价值与诊断决策错位的问题，提出MedEVM基准和EVD-Harness框架，通过证据验证控制提交，显著提升诊断准确性与可靠性。
AI 中文摘要
从不足或误导性证据中得出的正确诊断可能构成临床危害，然而基于结果的准确性可能会奖励这种幸运的猜测。我们将诊断决策与可用证据价值之间的这种不匹配称为证据-价值错位（EVM）。为了独立于诊断准确性来区分证据依据，我们引入了 MedEVM，一个动态基准测试环境，包含 24 个疾病系统中的 1,050 个病例。观察结果逐轮到达，要求模型通过决定是等待更多证据还是提交诊断来持续校准其决策。在 9 个 LLM 中，观察到了四个有趣的模式。（1）校准不当的证据跟踪。做出诊断往往无法校准证据充分性，即使在能力更强的模型中也是如此，并且在推理模式下甚至更糟。（2）错位的诊断提交。即使有充分证据，对正确诊断的信心也往往无法确保及时提交。（3）证据顺序很重要。即使模型置信度保持相似，重新排序相同的证据也会改变诊断结果。（4）误导性证据仍然具有影响力。即使先前的证据已经充分，添加的误导性证据也会改变诊断方向。我们进一步验证了 EVM 能预测错误，并且防止过早提交能提高准确性。这些发现催生了证据验证诊断框架（EVD-Harness）。它通过离线对比诊断维基和三个在线控制阶段（即观察管理、提案与证人验证、以及诊断提交控制）将诊断生成与提交解耦。在五个 LLM 中，EVD-Harness 将准确性提高了 12.0 至 51.1 个百分点，同时减轻了与 EVM 相关的失败。我们的结果表明，在提交前验证证据支持可以使诊断决策更加可靠。
英文摘要
A correct diagnosis reached from insufficient or misleading evidence can pose a clinical hazard, yet outcome-based accuracy may reward such lucky guesses. We call this mismatch between diagnostic decisions and the value of available evidence Evidence-Value Misalignment (EVM). To disentangle evidential grounding independently from diagnostic accuracy, we introduce MedEVM, a dynamic benchmarking environment comprising 1,050 cases across 24 disease systems. Observations arrive turn by turn, requiring models to continuously calibrate its decision by deciding whether to wait for more evidence or submit a diagnosis. Across 9 LLMs, four interesting patterns are observed. (1) Miscalibrated evidence tracking. Making a diagnosis often fails to calibrate evidence sufficiency, even in more capable models, and even worsens in reasoning mode. (2) Misaligned diagnosis submission. Confidence in the correct diagnosis often fails to ensure timely submission despite sufficient evidence. (3) Evidence order matters. Reordering the same evidence changes diagnoses even when model confidence remains similar. (4) Misleading evidence remains influential. Added misleading evidence redirects diagnoses even after prior evidence becomes sufficient. We further verify that EVM predicts errors and that preventing premature submission improves accuracy. These findings motivate Evidence-Verified Diagnosis Harness (EVD-Harness). It decouples diagnosis generation from submission through an offline Contrastive Diagnostic Wiki and three online control stages, namely observation management, proposal and witness verification, and diagnosis submission control. Across five LLMs, EVD-Harness improves accuracy by 12.0--51.1 percentage points while mitigating EVM-related failures. Our results demonstrate that verifying evidential support before submission can make diagnostic decisions more reliable.
Comments33 pages, 10 figures
