The rapid advancement of large language models(LLMs) has prompted significant interest in their potential applications in medical domains. This paper presents a comprehensive benchmark evaluation of 27 state-of-the-art LLMs on Chinese medical examination questions, encompassing seven medical specialties across two professional levels. We introduce a robust evaluation framework that assesses model performance on 2,800 carefully curated questions from cardiovascular, gastroenterology, hematology, infectious diseases, nephrology, neurology, and respiratory medicine domains. Our dataset distinguishes between attending physician and senior physician difficulty levels, providing nuanced insights into model capabilities across varying complexity. Our empirical analysis reveals substantial performance variations among models, with Mixtral-8x7B achieving the highest overall accuracy of 74.25%, followed by DeepSeek-R1-671B at 64.07%. Notably, we observe no consistent correlation between model size and performance, as evidenced by the strong performance of smaller mixture-of-experts architectures. The evaluation demonstrates significant performance gaps between medical specialties, with models generally performing better on cardiovascular and neurology questions compared to gastroenterology and nephrology domains. Furthermore, our analysis indicates minimal performance degradation between attending and senior physician levels for top-performing models, suggesting robust generalization capabilities. This benchmark provides critical insights for the deployment of LLMs in medical education and clinical decision support systems, highlighting both the promise and current limitations of these technologies in specialized medical contexts.
CPsyExam: A Chinese Benchmark for Evaluating Psychology using Examinations
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Jilin University(吉林大学);University of Science and Technology of China(中国科学技术大学);Shenzhen University of Advanced Technology(深圳理工大学);University of Macau(澳门大学);Huazhong University of Science and Technology(华中科技大学)
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
East China Normal University(华东师范大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Hasso Plattner Institute(哈索·普拉特纳研究所);University of Potsdam(波茨坦大学);Cooperative Medianet Innovation Center, Shanghai Jiao Tong University(上海交通大学合作媒体网络创新中心)
FoodSky: A Food-oriented Large Language Model that Passes the Chef and Dietetic Examination
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Institute of Intelligent Computing Technology, Chinese Academy of Sciences(中国科学院智能计算技术研究所);Zhongke Shenjian (Suzhou) Technology Co., Ltd(中科神箭(苏州)科技有限公司)
MITRA: A Large-Scale Parallel Corpus and Multilingual Pretrained Language Model for Machine Translation and Semantic Retrieval for Pāli, Sanskrit, Buddhist Chinese, and Tibetan
Center for Integrated Japanese Studies, Tohoku University(东京东京大学综合日语研究机构);University of California, Berkeley(加州大学伯克利分校);Berkeley Artificial Intelligence Research (BAIR)(伯克利人工智能研究(BAIR))
TCM-Eval: An Expert-Level Dynamic and Extensible Benchmark for Traditional Chinese Medicine
School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);Beijing Zhimingtang Technology Co., Ltd.(北京智明堂科技有限公司);Beijing Zhiyan AI Technology Co., Ltd.(北京智言AI科技有限公司);Guangzhou University of Chinese Medicine(广州中医药大学)
Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset
Harvard University(哈佛大学);Peking University(北京大学);Second Affiliated Hospital of Zhejiang University School of Medicine(浙江大学医学院附属第二医院);Johns Hopkins University(约翰斯·霍普金斯大学);Yale University(耶鲁大学);Tianyi Traffic Technology(天翼交通科技);Ant Group(蚂蚁集团);Boston Children’s Hospital(波士顿儿童医院);Harvard Business School(哈佛商学院);Alibaba Group(阿里巴巴集团);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Tianyi: A Traditional Chinese Medicine all-rounder language model and its Real-World Clinical Practice
Nanjing University of Chinese Medicine(南京中医药大学);Beijing University of Chinese Medicine(北京中医药大学);Capital Medical University(首都医科大学);Zhejiang Lab(之江实验室);China Academy of Chinese Medical Sciences(中国中医科学院)
DeepSeek-R1 Outperforms Gemini 2.0 Pro, OpenAI o1, and o3-mini in Bilingual Complex Ophthalmology Reasoning
School of Optometry, The Hong Kong Polytechnic University(香港理工大学视光学院);Zhejiang University(浙江大学);Research Centre for SHARP Vision (RCSV), The Hong Kong Polytechnic University(香港理工大学SHARP视觉研究中心);Centre for Eye and Vision Research (CEVR)(眼科与视觉研究中心)
Large Language Models Leverage External Knowledge to Extend Clinical Insight Beyond Language Boundaries
School of Public Health, Zhejiang University School of Medicine(浙江大学医学院公共卫生学院);Jarvis Research Center, Tencent YouTu Lab(腾讯优图实验室Jarvis研究中心);Harvard T.H. Chan School of Public Health(哈佛大学陈曾熙公共卫生学院);Harvard Medical School(哈佛医学院)
LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?
School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University(计算机学院、多媒体软件国家工程研究中心、人工智能研究院、多媒体与网络通信工程湖北省重点实验室、武汉大学)
Mastering Olympiad-Level Physics with Artificial Intelligence
School of Physics, Peking University(物理系,北京大学);School of Electronics Engineering and Computer Science, Peking University(电子工程与计算机科学系,北京大学);Center for High Energy Physics, Peking University(高能物理中心,北京大学)
Improving TCM Question Answering through Tree-Organized Self-Reflective Retrieval with LLMs
School of Basic Medical Sciences, Zhejiang Chinese Medical University(浙江中医药大学基础医学院);Breast Disease Specialist Hospital of Guangdong Provincial Hospital of Chinese Medicine(广东省中医院乳腺疾病专科医院);Chinese Medicine Artificial Intelligence Joint Engineering Center, Zhejiang Chinese Medical University(浙江中医药大学中医药人工智能联合工程中心);Faculty of Chinese Medicine, Macau University of Science and Technology(澳门科技大学中医药学院)