MedMT-Bench: LLMs能否在医学场景中记忆并理解长多轮对话？
MedMT-Bench: Can LLMs Memorize and Understand Long Multi-Turn Conversations in Medical Scenarios?
AI总结：
MedMT-Bench通过模拟诊断治疗流程，测试LLM在长上下文记忆、干扰鲁棒性和安全防御方面的表现，发现17种前沿模型均无法达到60%以上的准确率。
摘要：
大型语言模型（LLMs）在多个专业领域展现出卓越能力，并已应用于高风险领域如医学。然而，现有医学相关基准测试 rarely 挑战实际所需的长上下文记忆、干扰鲁棒性和安全防御能力。为此，我们引入MedMT-Bench，一个具有挑战性的医学多轮指令跟随基准测试，通过场景级数据合成并经人工专家编辑，生成400个测试案例，每个案例平均22轮（最多52轮），涵盖5种困难的指令跟随问题。评估中，我们提出LLM-as-judge协议，结合实例级评分标准和原子测试点，与专家注释验证后的人机一致率为91.94%。我们测试了17种前沿模型，所有模型在MedMT-Bench上的整体准确率均低于60.00%，最佳模型达到59.75%。MedMT-Bench可成为推动未来研究向更安全可靠医学AI发展的关键工具。该基准测试可在https://openreview.net/attachment?id=aKyBCsPOHB&name=supplementary_material获取。
英文摘要：
Large Language Models (LLMs) have demonstrated impressive capabilities across various specialist domains and have been integrated into high-stakes areas such as medicine. However, as existing medical-related benchmarks rarely stress-test the long-context memory, interference robustness, and safety defense required in practice. To bridge this gap, we introduce MedMT-Bench, a challenging medical multi-turn instruction following benchmark that simulates the entire diagnosis and treatment process. We construct the benchmark via scene-by-scene data synthesis refined by manual expert editing, yielding 400 test cases that are highly consistent with real-world application scenarios. Each test case has an average of 22 rounds (maximum of 52 rounds), covering 5 types of difficult instruction following issues. For evaluation, we propose an LLM-as-judge protocol with instance-level rubrics and atomic test points, validated against expert annotations with a human-LLM agreement of 91.94\%. We test 17 frontier models, all of which underperform on MedMT-Bench (overall accuracy below 60.00\%), with the best model reaching 59.75\%. MedMT-Bench can be an essential tool for driving future research towards safer and more reliable medical AI. The benchmark is available in https://openreview.net/attachment?id=aKyBCsPOHB&name=supplementary_material
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
