浅层稳健性，深层脆弱性：医学大语言模型的多轮评估
Shallow Robustness, Deep Vulnerabilities: Multi-Turn Evaluation of Medical LLMs
AI总结：
本文提出MedQA-Followup框架评估医学大语言模型多轮稳健性，发现模型在浅层扰动下尚可，但深层多轮交互中准确率大幅下降，间接上下文干预危害更大，凸显多轮稳健性对临床部署的关键性。
摘要：
大语言模型（LLMs）正迅速进入医学临床应用，但它们在现实多轮交互中的可靠性仍鲜为人知。现有评估框架通常评估理想化条件下的单轮问答，忽视了医疗咨询中常见的冲突输入、误导性上下文和权威影响等复杂性。我们提出了MedQA-Followup，一个系统评估医学问答中多轮稳健性的框架。我们的方法区分了浅层稳健性（抵抗误导性初始上下文）和深层稳健性（在答案跨轮次受到挑战时保持准确性），同时引入了一个间接-直接轴，将上下文框架（间接）与明确建议（直接）区分开来。通过在MedQA数据集上使用受控干预，我们评估了五个最先进的大语言模型，发现虽然模型在浅层扰动下表现良好，但在多轮设置中表现出严重脆弱性，Claude Sonnet 4的准确率从91.2%降至低至13.5%。与直觉相反，基于上下文的间接干预通常比直接建议更具危害性，导致各模型准确率下降更大，暴露了临床部署中的重大脆弱性。进一步的分层分析揭示了模型差异，一些模型在重复干预下出现额外性能下降，而其他模型部分恢复甚至改善。这些发现强调多轮稳健性是医学大语言模型安全可靠部署中一个关键但未充分探索的维度。
英文摘要：
Large language models (LLMs) are rapidly transitioning into medical clinical use, yet their reliability under realistic, multi-turn interactions remains poorly understood. Existing evaluation frameworks typically assess single-turn question answering under idealized conditions, overlooking the complexities of medical consultations where conflicting input, misleading context, and authority influence are common. We introduce MedQA-Followup, a framework for systematically evaluating multi-turn robustness in medical question answering. Our approach distinguishes between shallow robustness (resisting misleading initial context) and deep robustness (maintaining accuracy when answers are challenged across turns), while also introducing an indirect-direct axis that separates contextual framing (indirect) from explicit suggestion (direct). Using controlled interventions on the MedQA dataset, we evaluate five state-of-the-art LLMs and find that while models perform reasonably well under shallow perturbations, they exhibit severe vulnerabilities in multi-turn settings, with accuracy dropping from 91.2% to as low as 13.5% for Claude Sonnet 4. Counterintuitively, indirect, context-based interventions are often more harmful than direct suggestions, yielding larger accuracy drops across models and exposing a significant vulnerability for clinical deployment. Further compounding analyses reveal model differences, with some showing additional performance drops under repeated interventions while others partially recovering or even improving. These findings highlight multi-turn robustness as a critical but underexplored dimension for safe and reliable deployment of medical LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
