医疗对话中大语言模型错误的自动复制
Automatic Replication of LLM Mistakes in Medical Conversations
AI总结：
本文提出MedMistake自动管道，通过提取医疗对话中LLM的错误生成单次问答对，构建基准测试集，验证了GPT、Claude和Grok在医疗问答任务中的最佳性能。
摘要：
大型语言模型（LLMs）在临床环境中越来越多地使用多维评分标准进行评估，这些标准量化了推理质量、安全性和以患者为中心性。然而，复制其他LLM模型中的特定错误并不容易，通常需要手动努力。我们介绍了MedMistake，一个自动管道，该管道从患者-医生对话中提取LLM的错误，并将其转换为单次问答对的基准。我们的管道（1）创建了LLM患者和LLM医生之间的复杂对话数据，（2）由两名LLM评委在多种维度上进行评估，（3）从这些错误中创建了简化后的单次问答场景。我们发布了MedMistake-All数据集，其中包含3,390个单次问答对，目前GPT-5和Gemini 2.5 Pro无法正确回答，由两名LLM评委判断。我们使用医疗专家验证了211/3390个问题（MedMistake-Bench），并用其对12种前沿LLM进行了最终评估：Claude Opus 4.5、Claude Sonnet 4.5、DeepSeek-Chat、Gemini 2.5 Pro、Gemini 3 Pro、GPT-4o、GPT-5、GPT-5.1、GPT-5.2、Grok 4、Grok 4.1、Mistral Large。我们发现GPT模型、Claude和Grok在MedMistake-Bench上表现最佳。我们发布了由医生验证的基准（MedMistake-Bench）以及完整数据集（MedMistake-All）在https://huggingface.co/datasets/TheLumos/MedicalMistakeBenchmark。
英文摘要：
Large language models (LLMs) are increasingly evaluated in clinical settings using multi-dimensional rubrics which quantify reasoning quality, safety, and patient-centeredness. Yet, replicating specific mistakes in other LLM models is not straightforward and often requires manual effort. We introduce MedMistake, an automatic pipeline that extracts mistakes LLMs make in patient-doctor conversations and converts them into a benchmark of single-shot QA pairs. Our pipeline (1) creates complex, conversational data between an LLM patient and LLM doctor, (2) runs an evaluation with a committee of 2 LLM judges across a variety of dimensions and (3) creates simplified single-shot QA scenarios from those mistakes. We release MedMistake-All, a dataset of 3,390 single-shot QA pairs where GPT-5 and Gemini 2.5 Pro are currently failing to answer correctly, as judged by two LLM judges. We used medical experts to validate a subset of 211/3390 questions (MedMistake-Bench), which we used to run a final evaluation of 12 frontier LLMs: Claude Opus 4.5, Claude Sonnet 4.5, DeepSeek-Chat, Gemini 2.5 Pro, Gemini 3 Pro, GPT-4o, GPT-5, GPT-5.1, GPT-5.2, Grok 4, Grok 4.1, Mistral Large. We found that GPT models, Claude and Grok obtained the best performance on MedMistake-Bench. We release both the doctor-validated benchmark (MedMistake-Bench), as well as the full dataset (MedMistake-All) at https://huggingface.co/datasets/TheLumos/MedicalMistakeBenchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
