发表机构
Independent Researcher; School of Computation, Information and Technology, Technical University of Munich; Institute of General Practice, Faculty of Medicine and Medical Center, University of Freiburg（独立研究者; 慕尼黑工业大学计算、信息与技术学院; 弗莱堡大学医学中心医学院普通实践研究所）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究推出TAF-MED医疗安全基准，评估8个大语言模型的多轮医疗对话安全表现，发现多数模型会在后续对话中出现安全拒绝崩溃，自动评判工具与医生标注一致性较高，将公开基准支持相关研究。
AI 中文摘要
大语言模型（LLMs）越来越多地提供可能影响治疗决策的对话式健康信息，但现有基准无法明确在明确自我治疗意图后，药物安全边界是否会在后续对话中持续存在。我们推出TAF-MED，这是一个经医生审核的基准，包含500个固定三轮场景，并在4000次对话中评估了8个大语言模型。基于评分标准的自动评判器将响应标记为SAFE（安全）、LEAKY（有风险）或UNSAFE（不安全），两名医生独立标注了模型均衡的400次对话随机子集。我们评估了不安全指导、严格安全初始响应后的崩溃情况以及模型排名稳定性。总体而言，71.6%的对话包含不安全响应，且在以严格安全响应开始的对话中，61.4%后续崩溃为不安全；模型级别的崩溃率范围为24.4%至96.2%。28个模型对中有4个在初始不安全率与崩溃率之间排名反转。自动标签与经医生裁定的参考结果达成94.3%的一致性（κ=0.895）。这些发现表明，首轮安全并非对话安全持续性的完整替代指标，并推动对完整对话轨迹的评估。我们将在Hugging Face上发布TAF-MED，以支持多轮医疗安全的可复现研究。
英文摘要
Large language models (LLMs) increasingly provide conversational health information that may influence treatment decisions, yet existing benchmarks do not isolate whether medication-safety boundaries persist across follow-ups after explicit self-treatment intent. We introduce TAF-MED, a physician-reviewed benchmark of 500 fixed three-turn scenarios, and evaluate eight LLMs across 4,000 conversations. A rubric-based automated judge labelled responses as SAFE, LEAKY, or UNSAFE, and two physicians independently annotated a model-balanced random subset of 400 conversations. We assessed unsafe guidance, collapse after a strictly SAFE initial response, and model-ranking stability. Overall, 71.6% of conversations contained an UNSAFE response, and 61.4% of those beginning with a strictly SAFE response later collapsed to UNSAFE; model-level collapse rates ranged from 24.4% to 96.2%. Four of 28 model pairs reversed order between initial unsafe and collapse rates. Automated labels achieved 94.3% agreement with the adjudicated physician reference ($κ= 0.895$). These findings show that first-turn safety is an incomplete proxy for conversational safety persistence and motivate evaluation across complete dialogue trajectories. We will release TAF-MED on Hugging Face to support reproducible research on multi-turn medical safety.
