LiveMedBench: 一个无污染的医疗基准测试，用于具有自动评分评估的LLM
LiveMedBench: A Contamination-Free Medical Benchmark for LLMs with Automated Rubric Evaluation
AI总结：
LiveMedBench通过持续更新和自动评分框架，提供无污染的医疗基准测试，验证LLM在临床推理中的性能，揭示数据污染和上下文适应性问题。
摘要：
大型语言模型（LLM）在高风险临床环境中的部署需要严格且可靠的评估。然而，现有的医疗基准测试仍然静态，存在两个关键限制：（1）数据污染，测试集意外泄露到训练语料库，导致性能估计被高估；（2）时间错位，无法捕捉医学知识的快速演变。此外，当前用于开放式临床推理的评估指标往往依赖于浅层词汇重叠（例如ROUGE）或主观LLM作为评判者评分，这两种方法都不足以验证临床正确性。为了弥补这些差距，我们引入了LiveMedBench，一个持续更新、无污染且基于评分的基准测试，每周从在线医学社区中收集现实世界的临床案例，确保严格的时间分离与模型训练数据。我们提出了一种多代理临床筛选框架，用于过滤原始数据噪声并验证临床完整性，依据循证医学原则。在评估方面，我们开发了自动评分评估框架，将医生的回应分解为细粒度、案例特定的准则，其与专家医生的对齐程度明显优于LLM作为评判者。截至目前，LiveMedBench包含2,756个现实世界案例，涵盖38个医学专业和多种语言，并配以16,702个独特的评估准则。对38个LLM的广泛评估显示，即使表现最好的模型也仅达到39.2%，且84%的模型在截止后案例上表现出性能下降，证实了普遍的数据污染风险。错误分析进一步发现，上下文应用而非事实知识是主要瓶颈，35-48%的失败源于无法将医学知识适配到患者特定约束。
英文摘要：
The deployment of Large Language Models (LLMs) in high-stakes clinical settings demands rigorous and reliable evaluation. However, existing medical benchmarks remain static, suffering from two critical limitations: (1) data contamination, where test sets inadvertently leak into training corpora, leading to inflated performance estimates; and (2) temporal misalignment, failing to capture the rapid evolution of medical knowledge. Furthermore, current evaluation metrics for open-ended clinical reasoning often rely on either shallow lexical overlap (e.g., ROUGE) or subjective LLM-as-a-Judge scoring, both inadequate for verifying clinical correctness. To bridge these gaps, we introduce LiveMedBench, a continuously updated, contamination-free, and rubric-based benchmark that weekly harvests real-world clinical cases from online medical communities, ensuring strict temporal separation from model training data. We propose a Multi-Agent Clinical Curation Framework that filters raw data noise and validates clinical integrity against evidence-based medical principles. For evaluation, we develop an Automated Rubric-based Evaluation Framework that decomposes physician responses into granular, case-specific criteria, achieving substantially stronger alignment with expert physicians than LLM-as-a-Judge. To date, LiveMedBench comprises 2,756 real-world cases spanning 38 medical specialties and multiple languages, paired with 16,702 unique evaluation criteria. Extensive evaluation of 38 LLMs reveals that even the best-performing model achieves only 39.2%, and 84% of models exhibit performance degradation on post-cutoff cases, confirming pervasive data contamination risks. Error analysis further identifies contextual application-not factual knowledge-as the dominant bottleneck, with 35-48% of failures stemming from the inability to tailor medical knowledge to patient-specific constraints.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
