arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.03030cs.CL
收藏

在CMExam上对大型语言模型进行基准测试——一个全面的中文医学考试数据集

Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset

Harvard University(哈佛大学);Peking University(北京大学);Second Affiliated Hospital of Zhejiang University School of Medicine(浙江大学医学院附属第二医院);Johns Hopkins University(约翰斯·霍普金斯大学);Yale University(耶鲁大学);Tianyi Traffic Technology(天翼交通科技);Ant Group(蚂蚁集团);Boston Children’s Hospital(波士顿儿童医院);Harvard Business School(哈佛商学院);Alibaba Group(阿里巴巴集团);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))

Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, Michael Lingzhi Li

AI总结:

本文提出CMExam,一个包含6万+题目及医学注释的中文医学考试数据集,用于评估LLMs,发现GPT-4准确率61.6%仍低于人类71.6%,解释任务也有改进空间。

摘要:

大型语言模型(LLMs)的最新进展改变了问答(QA)领域。然而,由于缺乏标准化和全面的数据集,在医学领域评估LLMs具有挑战性。为了填补这一空白,我们引入了CMExam,其数据来源于中国国家医师资格考试。CMExam包含6万多个多项选择题,用于标准化和客观的评估,以及用于以开放式方式评估模型推理的解答解释。为了对LLMs进行深入分析,我们邀请了医学专业人员标注五个额外的问题级注释,包括疾病组、临床科室、医学学科、能力领域和问题难度级别。除了数据集之外,我们还在CMExam上使用代表性的LLMs和QA算法进行了彻底的实验。结果显示,GPT-4的最高准确率为61.6%,加权F1分数为0.617。与人类71.6%的准确率相比,这些结果突显了巨大的差距。对于解释任务,虽然LLMs能够生成相关的推理并在微调后表现出性能提升,但它们仍未达到理想标准,表明有充足的改进空间。据我们所知,CMExam是第一个提供全面医学注释的中文医学考试数据集。LLM评估的实验和发现也为开发中文医学QA系统和LLM评估流程中的挑战和潜在解决方案提供了有价值的见解。数据集和相关代码可在https://github.com/williamliujl/CMExam获取。

英文摘要:

Recent advancements in large language models (LLMs) have transformed the field of question answering (QA). However, evaluating LLMs in the medical field is challenging due to the lack of standardized and comprehensive datasets. To address this gap, we introduce CMExam, sourced from the Chinese National Medical Licensing Examination. CMExam consists of 60K+ multiple-choice questions for standardized and objective evaluations, as well as solution explanations for model reasoning evaluation in an open-ended manner. For in-depth analyses of LLMs, we invited medical professionals to label five additional question-wise annotations, including disease groups, clinical departments, medical disciplines, areas of competency, and question difficulty levels. Alongside the dataset, we further conducted thorough experiments with representative LLMs and QA algorithms on CMExam. The results show that GPT-4 had the best accuracy of 61.6% and a weighted F1 score of 0.617. These results highlight a great disparity when compared to human accuracy, which stood at 71.6%. For explanation tasks, while LLMs could generate relevant reasoning and demonstrate improved performance after finetuning, they fall short of a desired standard, indicating ample room for improvement. To the best of our knowledge, CMExam is the first Chinese medical exam dataset to provide comprehensive medical annotations. The experiments and findings of LLM evaluation also provide valuable insights into the challenges and potential solutions in developing Chinese medical QA systems and LLM evaluation pipelines. The dataset and relevant code are available at https://github.com/williamliujl/CMExam.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑