MLB：面向临床应用的大型语言模型评估场景驱动基准
MLB: A Scenario-Driven Benchmark for Evaluating Large Language Models in Clinical Applications
AI总结：
MLB基准通过场景驱动的评估方法，评估大型语言模型在临床应用中的表现，揭示模型在结构化任务与患者交互场景间的性能差异。
摘要：
大型语言模型（LLMs）的普及为医疗领域带来了变革性潜力，但实际部署受到缺乏评估真实临床效用的框架的阻碍。现有基准测试仅测试静态知识，无法捕捉临床实践中所需的动态、应用导向的能力。为此，我们引入了一个医学LLM基准（MLB），该基准全面评估LLM在基础知识和基于场景的推理方面的表现。MLB围绕五个核心维度构建：医学知识（MedKQA）、安全与伦理（MedSE）、医学记录理解（MedRU）、智能服务（SmartServ）和智能医疗（SmartCare）。该基准整合了22个数据集（17个新整理的）来自多样的中文临床来源，涵盖64个临床专科。其设计特征是严谨的整理流程，涉及300名执业医师。此外，我们还提供了一种可扩展的评估方法，核心是通过监督微调（SFT）在专家注释上训练的专用判断模型。我们对10个领先模型的全面评估揭示了一个关键的转化差距：虽然排名第一的模型Kimi-K2-Instruct（总体准确率77.3%）在结构化任务如信息提取（在MedRU中准确率87.8%）方面表现出色，但在面向患者的情景中表现却骤降（在SmartServ中为61.3%）。此外，较小的Baichuan-M2-32B在安全评分（在MedSE中为90.6%）方面表现出色，这表明针对性训练同样至关重要。我们的专用判断模型，通过在19,000个专家注释的医疗数据集上进行SFT训练，实现了92.1%的准确率，F1分数为94.37%，Cohen's Kappa为81.3%的人工-人工智能一致性，验证了可重复和专家对齐的评估协议。MLB因此提供了一个严谨的框架，以指导开发具有临床可行性的LLMs。
英文摘要：
The proliferation of Large Language Models (LLMs) presents transformative potential for healthcare, yet practical deployment is hindered by the absence of frameworks that assess real-world clinical utility. Existing benchmarks test static knowledge, failing to capture the dynamic, application-oriented capabilities required in clinical practice. To bridge this gap, we introduce a Medical LLM Benchmark MLB, a comprehensive benchmark evaluating LLMs on both foundational knowledge and scenario-based reasoning. MLB is structured around five core dimensions: Medical Knowledge (MedKQA), Safety and Ethics (MedSE), Medical Record Understanding (MedRU), Smart Services (SmartServ), and Smart Healthcare (SmartCare). The benchmark integrates 22 datasets (17 newly curated) from diverse Chinese clinical sources, covering 64 clinical specialties. Its design features a rigorous curation pipeline involving 300 licensed physicians. Besides, we provide a scalable evaluation methodology, centered on a specialized judge model trained via Supervised Fine-Tuning (SFT) on expert annotations. Our comprehensive evaluation of 10 leading models reveals a critical translational gap: while the top-ranked model, Kimi-K2-Instruct (77.3% accuracy overall), excels in structured tasks like information extraction (87.8% accuracy in MedRU), performance plummets in patient-facing scenarios (61.3% in SmartServ). Moreover, the exceptional safety score (90.6% in MedSE) of the much smaller Baichuan-M2-32B highlights that targeted training is equally critical. Our specialized judge model, trained via SFT on a 19k expert-annotated medical dataset, achieves 92.1% accuracy, an F1-score of 94.37%, and a Cohen's Kappa of 81.3% for human-AI consistency, validating a reproducible and expert-aligned evaluation protocol. MLB thus provides a rigorous framework to guide the development of clinically viable LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
