arXiv 2312.128062023-12-20cs.CLcs.AI

MedBench：用于评估医学大语言模型的大规模中文基准

MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models

East China Normal University（华东师范大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Hasso Plattner Institute（哈索·普拉特纳研究所）；University of Potsdam（波茨坦大学）；Cooperative Medianet Innovation Center, Shanghai Jiao Tong University（上海交通大学合作媒体网络创新中心）

Yan Cai, Linlin Wang, Ye Wang, Gerard de Melo, Ya Zhang, Yanfeng Wang, Liang He

备注：: accepted by AAAI-24

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对医学大语言模型缺乏统一评估标准的问题，提出包含40,041个问题的中文医学基准MedBench，涵盖资格考试与临床病例，实验发现中文医学模型表现欠佳，而通用模型具备一定医学知识。

摘要：

医学领域中各种医学大语言模型（LLM）的出现，凸显了对统一评估标准的需求，因为对LLM进行人工评估被证明既耗时又费力。为解决这一问题，我们引入了MedBench，一个面向中文医学领域的综合性基准，包含来自不同医学分支的真实考试习题和医学报告的40,041个问题。具体而言，该基准由四个关键部分组成：中国执业医师资格考试、住院医师规范化培训考试、主治医师资格考试，以及涵盖检查、诊断和治疗的真实临床病例。MedBench模拟了中国大陆医生的教育进阶和临床实践经历，从而使其成为评估医学语言学习模型知识掌握程度和推理能力的可信基准。我们进行了大量实验，并从多个角度进行了深入分析，最终得出以下发现：（1）中文医学LLM在此基准上表现不佳，凸显了在临床知识和诊断准确性方面需要重大进展。（2）若干通用领域LLM意外地拥有相当可观的医学知识。这些发现阐明了LLM在MedBench背景下的能力与局限性，最终目标是助力医学研究社区。

英文摘要：

The emergence of various medical large language models (LLMs) in the medical domain has highlighted the need for unified evaluation standards, as manual evaluation of LLMs proves to be time-consuming and labor-intensive. To address this issue, we introduce MedBench, a comprehensive benchmark for the Chinese medical domain, comprising 40,041 questions sourced from authentic examination exercises and medical reports of diverse branches of medicine. In particular, this benchmark is composed of four key components: the Chinese Medical Licensing Examination, the Resident Standardization Training Examination, the Doctor In-Charge Qualification Examination, and real-world clinic cases encompassing examinations, diagnoses, and treatments. MedBench replicates the educational progression and clinical practice experiences of doctors in Mainland China, thereby establishing itself as a credible benchmark for assessing the mastery of knowledge and reasoning abilities in medical language learning models. We perform extensive experiments and conduct an in-depth analysis from diverse perspectives, which culminate in the following findings: (1) Chinese medical LLMs underperform on this benchmark, highlighting the need for significant advances in clinical knowledge and diagnostic precision. (2) Several general-domain LLMs surprisingly possess considerable medical knowledge. These findings elucidate both the capabilities and limitations of LLMs within the context of MedBench, with the ultimate goal of aiding the medical research community.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。