arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.21701cs.CLcs.LG
收藏

47B混合专家模型在中文医学考试中超越671B密集模型

47B Mixture-of-Experts Beats 671B Dense Models on Chinese Medical Examinations

AI Agent Lab, Vokram Group(AI代理实验室,Vokram集团);Purdue University(普渡大学);University of Minnesota(明尼苏达大学);Imperial College London(伦敦帝国理工学院)

Chiung-Yi Tseng, Danyang Zhang, Tianyang Wang, Hongying Luo, Lu Chen, Junming Huang, Jibin Guan, Junfeng Hao, Junhao Song, Xinyuan Song, Ziqian Bi

AI总结:

本文评估了多种大语言模型在中文医学考试中的表现,发现47B混合专家模型在准确率上超越了671B密集模型,揭示了模型大小与性能无直接关联,并指出了不同医学专科间性能差异及模型泛化能力。

摘要:

大语言模型(LLMs)的快速发展引发了其在医疗领域潜在应用的广泛关注。本文对27种最先进的LLMs在中文医学考试问题上的综合基准评估,涵盖两个专业层次的七个医学专科。我们引入了一个稳健的评估框架,评估模型在心血管、胃肠病学、血液病学、传染病学、肾病学、神经病学和呼吸病学领域的2800个精心挑选的问题上的性能。我们的数据集区分了住院医师和副主任医师难度等级,为模型在不同复杂性下的能力提供了细致的洞察。我们的实证分析揭示了模型之间显著的性能差异,其中Mixtral-8x7B实现了最高的整体准确率74.25%,其次是DeepSeek-R1-671B的64.07%。值得注意的是,我们观察到模型大小与性能之间没有一致的相关性,这体现在较小的混合专家架构上表现强劲。评估显示医学专科之间存在显著的性能差距,模型在心血管和神经病学问题上的表现优于胃肠病学和肾病学领域。此外,我们的分析表明,顶级表现模型在住院医师和副主任医师等级之间几乎没有性能下降,表明了强大的泛化能力。该基准为LLMs在医疗教育和临床决策支持系统中的部署提供了关键见解,突显了这些技术在专门医疗领域中的潜力和当前的局限性。

英文摘要:

The rapid advancement of large language models(LLMs) has prompted significant interest in their potential applications in medical domains. This paper presents a comprehensive benchmark evaluation of 27 state-of-the-art LLMs on Chinese medical examination questions, encompassing seven medical specialties across two professional levels. We introduce a robust evaluation framework that assesses model performance on 2,800 carefully curated questions from cardiovascular, gastroenterology, hematology, infectious diseases, nephrology, neurology, and respiratory medicine domains. Our dataset distinguishes between attending physician and senior physician difficulty levels, providing nuanced insights into model capabilities across varying complexity. Our empirical analysis reveals substantial performance variations among models, with Mixtral-8x7B achieving the highest overall accuracy of 74.25%, followed by DeepSeek-R1-671B at 64.07%. Notably, we observe no consistent correlation between model size and performance, as evidenced by the strong performance of smaller mixture-of-experts architectures. The evaluation demonstrates significant performance gaps between medical specialties, with models generally performing better on cardiovascular and neurology questions compared to gastroenterology and nephrology domains. Furthermore, our analysis indicates minimal performance degradation between attending and senior physician levels for top-performing models, suggesting robust generalization capabilities. This benchmark provides critical insights for the deployment of LLMs in medical education and clinical decision support systems, highlighting both the promise and current limitations of these technologies in specialized medical contexts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑