Mastering medical knowledge is crucial for medical-specific LLMs. However, despite the existence of medical benchmarks like MedQA, a unified framework that fully leverages existing knowledge bases to evaluate LLMs' mastery of medical knowledge is still lacking. In the study, we propose a novel framework PretexEval that dynamically generates reliable and diverse test samples to evaluate LLMs for any given medical knowledge base. We notice that test samples produced directly from knowledge bases by templates or LLMs may introduce factual errors and also lack diversity. To address these issues, we introduce a novel schema into our proposed evaluation framework that employs predicate equivalence transformations to produce a series of variants for any given medical knowledge point. Finally, these produced predicate variants are converted into textual language, resulting in a series of reliable and diverse test samples to evaluate whether LLMs fully master the given medical factual knowledge point. Here, we use our proposed framework to systematically investigate the mastery of medical factual knowledge of 12 well-known LLMs, based on two knowledge bases that are crucial for clinical diagnosis and treatment. The evaluation results illustrate that current LLMs still exhibit significant deficiencies in fully mastering medical knowledge, despite achieving considerable success on some famous public benchmarks. These new findings provide valuable insights for developing medical-specific LLMs, highlighting that current LLMs urgently need to strengthen their comprehensive and in-depth mastery of medical knowledge before being applied to real-world medical scenarios.
Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving
Department of Electronic Engineering, Tsinghua University, Beijing, China(清华大学电子工程系);College of AI, Tsinghua University, Beijing, China(清华大学人工智能学院);School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China(北京邮电大学计算机科学学院);iFLYTEK Research, Hefei, China(iFLYTEK研究院)
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
East China Normal University(华东师范大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Hasso Plattner Institute(哈索·普拉特纳研究所);University of Potsdam(波茨坦大学);Cooperative Medianet Innovation Center, Shanghai Jiao Tong University(上海交通大学合作媒体网络创新中心)
PACT: Learning Diverse Diagnostic Strategies via Privileged Synthesis and Branch Consensus
Beihang University(北京航空航天大学);Baidu(百度);ByteDance(字节跳动);Beijing Academy of Blockchain and Edge Computing(北京区块链与边缘计算研究院);Renmin University of China(中国人民大学)
AlpaCare:Instruction-tuned Large Language Models for Medical Application
University of California, Santa Barbara(加州大学圣巴巴拉分校);Chinese Academy of Medical Sciences and Peking Union Medical College(中国医学科学院北京协和医学院);University of Maryland, College Park(马里兰大学帕克分校)
PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications
Academy for Engineering and Technology, Fudan University(复旦大学工程与技术学院);Tencent Youtu Lab(腾讯优图实验室);Cognition and Intelligent Technology Laboratory(认知与智能技术实验室);Engineering Research Center of AI and Robotics, Ministry of Education(教育部人工智能与机器人工程研究中心);AI and Unmanned Systems Engineering Research Center of Jilin Province(吉林省人工智能与无人系统工程研究中心)
RAG-PRISM: A Personalized, Rapid, and Immersive Skill Mastery Framework with Adaptive Retrieval-Augmented Tutoring
College of Information Science, University of Arizona, Tucson, AZ, USA(信息科学学院,亚利桑那大学);Department of Electrical and Computer Engineering, University of Arizona, Tucson, AZ, USA(电气与计算机工程系,亚利桑那大学);Department of Systems and Industrial Engineering, University of Arizona, Tucson, AZ, USA(系统与工业工程系,亚利桑那大学);Department of Industrial Engineering, University of Sonora, Hermosillo, Mexico(工业工程系,索诺拉大学);Leonhard Center for Enhancement of Engineering Education, The Pennsylvania State University, University Park, PA, USA(工程教育增强中心,宾夕法尼亚州立大学)
MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks
National Engineering Research Center for Software Engineering, Peking University(北京大学国家软件工程研究中心);School of Computing and Data Science, The University of Hong Kong(香港大学计算与数据科学学院);Centre for Medical Informatics, The University of Edinburgh(爱丁堡大学医学信息学中心);ETH Zurich(苏黎世联邦理工学院);Health Data Research UK(英国健康数据研究所);Key Laboratory of High Confidence Software Technologies, Ministry of Education(教育部高可信软件技术重点实验室)
MedDiT: A Knowledge-Controlled Diffusion Transformer Framework for Dynamic Medical Image Generation in Virtual Simulated Patient
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Pattern Recognition Center, WeChat AI, Tencent Inc(腾讯公司微信人工智能模式识别中心);School of Computer Science, Wuhan University(武汉大学计算机学院)
TCM-3CEval: A Triaxial Benchmark for Assessing Responses from Large Language Models in Traditional Chinese Medicine
School of Acupuncture-Moxibustion and Tuina, Shanghai University of Traditional Chinese Medicine(上海中医药大学针灸推拿学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);China Academy of Chinese Medical Sciences(中国中医科学院)
Learning to Be A Doctor: Searching for Effective Medical Agent Architectures
AGI Lab, Westlake University(西拉雅大学AGI实验室);Henan University(河南大学);Affiliated Hospital of Xuzhou Medical University(徐州医科大学附属医院);Xuzhou Medical University(徐州医科大学);Nanyang Technological University(南洋理工大学);IHPC, Agency for Science, Technology and Research, Singapore(新加坡科技研究局IHPC);CFAR, Agency for Science, Technology and Research, Singapore(新加坡科技研究局CFAR)
Towards Autonomous and Auditable Medical Imaging Model Development
The Chinese University of Hong Kong(香港中文大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Microsoft Research(微软研究院);Lehigh University(里海大学)
MEPNet: Medical Entity-balanced Prompting Network for Brain CT Report Generation
College of Computer Science, Beijing University of Technology(北京工业大学计算机学院);School of Computing and Data Science, The University of Hong Kong(香港大学计算与数据科学学院)
A Comparative Study of Open-Source Large Language Models, GPT-4 and Claude 2: Multiple-Choice Test Taking in Nephrology
Pepperdine University(佩珀代因大学);Keck Data Science Institute(凯克数据科学研究所);David Geffen School of Medicine, University of California, Los Angeles(加州大学洛杉矶分校大卫·格芬医学院);Brain Research Institute, University of California, Los Angeles(加州大学洛杉矶分校脑研究所)
A Generalist Model for Diverse Text-Guided Medical Image Synthesis
Department of Cardiothoracic Surgery(心脏外科部门);Stanford Medicine(斯坦福医学);Department of Ophthalmology(眼科部门);Division of Cardiovascular Surgery(心血管外科分会);Penn Medicine(宾夕法尼亚医学)
Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias
The Ohio State University(俄亥俄州立大学);Imperial College London(伦敦帝国学院);The Hong Kong University of Science and Technology(香港科技大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Peking University(北京大学)
P-Transformer: A Prompt-based Multimodal Transformer Architecture For Medical Tabular Data
Saw Swee Hock School of Public Health, National University of Singapore(新加坡国立大学苏瑞福公共卫生学院);Institute of Data Science, National University of Singapore(新加坡国立大学数据科学研究院);Singapore General Hospital(新加坡综合医院)