相同结论，不同原因：LLM作为法官与临床医生在医疗聊天机器人完整性上的分歧
Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completeness
AI总结：
研究评估了LLM作为法官在检测医疗聊天机器人完整性上的可靠性，发现其与临床医生在判断标准上存在根本差异，无法有效替代人工评估。
摘要：
LLM-as-a-Judge框架正被越来越多地用于自动化评估，以替代人类专家。然而，在高风险医疗情境中其可靠性仍未得到证实。我们对检测不完整的患者面向医疗响应进行了压力测试，评估了三种细粒度的评分标准（通用-李克特、分析-评分表、动态-检查表）和三种基础模型，涵盖两个临床标注的数据集，包括HealthBench，这是最大的公开可用的医疗响应评估基准。LLM法官在区分完整和不完整响应方面在接近偶然率（AUC 0.49-0.66）至略高于偶然率范围内；在需要召回90%不完整响应的阈值下，临床医生仍需审查大部分数据集，提供无筛选效用。即使模型和临床医生的判断一致，他们也很少引用相同的解释；当他们分歧时，假阳性源于过度标记非必要的缺口，而假阴性反映的是彻底的检测失败。这些结果揭示了LLM法官和临床医生应用了根本不同的完整性标准；这一发现削弱了它们作为临床环境中自动评估或筛选过滤器的使用。
英文摘要：
LLM-as-a-Judge frameworks are increasingly trusted to automate evaluation in place of human experts, yet their reliability in high-stakes medical contexts remains unproven. We stress-test this assumption for detecting incomplete patient-facing medical responses, evaluating three rubric granularities (General-Likert, Analytical-Rubric, Dynamic-Checklist) and three backbone models across two clinician-annotated datasets, including HealthBench, the largest publicly available benchmark for medical response evaluation. LLM Judges discriminate complete from incomplete responses at and slightly above near chance (AUC $0.49$--$0.66$); at the threshold required to recall $90\%$ of incomplete responses, clinicians must still review the vast majority of the dataset, offering no triage utility. Even when model and clinician verdicts agree, they rarely cite the same explanation; and when they diverge, false positives stem from over-flagging non-essential gaps while false negatives reflect outright detection failures. These results reveal that LLM Judges and clinicians apply fundamentally different completeness standards; a finding that undermines their use as autonomous evaluators or triage filters in clinical settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
