LiveClin: 一种无泄漏的实时临床基准
LiveClin: A Live Clinical Benchmark without Leakage
AI总结：
LiveClin通过实时更新的临床病例基准，评估医学大语言模型在真实临床场景中的表现，揭示其在复杂医疗任务中的挑战。
摘要：
医学大语言模型评估的可靠性受到数据污染和知识过时的严重削弱，导致静态基准测试得分虚高。为解决这些问题，我们引入了LiveClin，一种用于近似真实临床实践的实时基准。该基准由当前、同行评审的病例报告构建，并每年更新两次，确保临床时效性并抵抗数据污染。通过经过验证的AI-人类工作流程，涉及239名医生，我们将真实的患者病例转化为复杂的多模态评估场景，覆盖整个临床流程。该基准目前包含1,407个病例报告和6,605个问题。我们在LiveClin上评估了26个模型，发现这些真实世界场景的难度非常大，表现最好的模型仅达到35.7%的病例准确率。在与人类专家对比时，主任医师取得了最高准确率，随后是住院医师，两者均超过了大多数模型。因此，LiveClin提供了一个持续演变、临床导向的框架，以指导医学大语言模型的发展，缩小这一差距，实现更高的可靠性和实际应用价值。我们的数据和代码在https://github.com/AQ-MedAI/LiveClin上公开。
英文摘要：
The reliability of medical LLM evaluation is critically undermined by data contamination and knowledge obsolescence, leading to inflated scores on static benchmarks. To address these challenges, we introduce LiveClin, a live benchmark designed for approximating real-world clinical practice. Built from contemporary, peer-reviewed case reports and updated biannually, LiveClin ensures clinical currency and resists data contamination. Using a verified AI-human workflow involving 239 physicians, we transform authentic patient cases into complex, multimodal evaluation scenarios that span the entire clinical pathway. The benchmark currently comprises 1,407 case reports and 6,605 questions. Our evaluation of 26 models on LiveClin reveals the profound difficulty of these real-world scenarios, with the top-performing model achieving a Case Accuracy of just 35.7%. In benchmarking against human experts, Chief Physicians achieved the highest accuracy, followed closely by Attending Physicians, with both surpassing most models. LiveClin thus provides a continuously evolving, clinically grounded framework to guide the development of medical LLMs towards closing this gap and achieving greater reliability and real-world utility. Our data and code are publicly available at https://github.com/AQ-MedAI/LiveClin.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
