Towards Evaluating and Building Versatile Large Language Models for Medicine
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);China Mobile Communications Group Co., Ltd.(中国移动通信集团有限公司);China Mobile Communications Group Shanghai Co., Ltd.(中国移动通信集团上海有限公司)
Chaoyi Wu, Pengcheng Qiu, Jinxin Liu, Hongfei Gu, Na Li, Ya Zhang, Yanfeng Wang, Weidi Xie
In this study, we present MedS-Bench, a comprehensive benchmark designed to evaluate the performance of large language models (LLMs) in clinical contexts. Unlike existing benchmarks that focus on multiple-choice question answering, MedS-Bench spans 11 high-level clinical tasks, including clinical report summarization, treatment recommendations, diagnosis, named entity recognition, and medical concept explanation, among others. We evaluated six leading LLMs, e.g., MEDITRON, Mistral, InternLM 2, Llama 3, GPT-4, and Claude-3.5 using few-shot prompting, and found that even the most sophisticated models struggle with these complex tasks. To address these limitations, we developed MedS-Ins, a large-scale instruction tuning dataset for medicine. MedS-Ins comprises 58 medically oriented language corpora, totaling 13.5 million samples across 122 tasks. To demonstrate the dataset's utility, we conducted a proof-of-concept experiment by performing instruction tuning on a lightweight, open-source medical language model. The resulting model, MMedIns-Llama 3, significantly outperformed existing models across nearly all clinical tasks. To promote further advancements in the application of LLMs to clinical challenges, we have made the MedS-Ins dataset fully accessible and invite the research community to contribute to its expansion.Additionally, we have launched a dynamic leaderboard for MedS-Bench, which we plan to regularly update the test set to track progress and enhance the adaptation of general LLMs to the medical domain. Leaderboard: https://henrychur.github.io/MedS-Bench/. Github: https://github.com/MAGIC-AI4Med/MedS-Ins.
MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine
School of Software Engineering, Sun Yat-sen University(中山大学软件学院);Institute of TCM Diagnostics, Hunan University of Chinese Medicine(湖南中医药大学中医诊断研究所);School of Artificial Intelligence, Guilin University of Electronic Technology(桂林电子科技大学人工智能学院);Department of Computer Science, Cornell University(康奈尔大学计算机科学系)
TCMBench: A Comprehensive Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine
Shanghai Institute of AI for Education, East China Normal University(华东师范大学人工智能教育研究院);School of Pharmacy, East China Normal University(华东师范大学药学院);Shanghai Skin Disease Hospital, School of Medicine, Tongji University(同济大学医学院附属上海市皮肤病医院)
KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations
Kyoto University(京都大学);The Hong Kong University of Science and Technology(香港科技大学);Xi’an Jiaotong-Liverpool University(西安交通大学利物浦大学);The University of Tokyo(东京大学)
RiTeK: A Dataset for Large Language Models Complex Reasoning over Textual Knowledge Graphs in Medicine
University of Connecticut(康涅狄格大学);University of Massachusetts, Amherst(马萨诸塞大学阿默斯特分校);School of Computing, and Augmented Intelligence, Arizona State University(亚利桑那州立大学计算与增强智能学院);UMass Chan Medical School(马萨诸塞大学陈医学院);University of Minnesota(明尼苏达大学);Rollins School of Public Health, Emory University(埃默里大学罗林斯公共卫生学院);Optum AI;University of Massachusetts, Lowell(马萨诸塞大学洛厄尔分校)
Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence
The University of Texas at Austin(德克萨斯大学奥斯汀分校);Northeastern University(东北大学);MD Anderson Cancer Center(MD安德森癌症中心);Georgia Institute of Technology(佐治亚理工学院)
Evaluating large language models in medical applications: a survey
The Hong Kong Polytechnic University(香港理工大学);Research Centre for SHARP Vision (RCSV), The Hong Kong Polytechnic University(香港理工大学SHARP视觉研究中心);Centre for Eye and Vision Research (CEVR)(眼与视觉研究中心);Renji Hospital, School of Medicine, Shanghai Jiao Tong University(上海交通大学医学院附属仁济医院);Perception Vision Medical Technologies Co. Ltd.(感知视觉医疗科技有限公司);The Hong Kong University of Science and Technology(香港科技大学)
FarsEval-PKBETS: A new diverse benchmark for evaluating Persian large language models
Shahid Beheshti University(沙希德·贝赫什提大学);Amirkabir University of Technology(阿米尔卡比尔理工大学);Qom University(库姆大学);Islamic Azad University(伊斯兰阿扎德大学);Sharif University of Technology(谢里夫理工大学)
MedCalc-Bench: Evaluating Large Language Models for Medical Calculations
National Library of Medicine, National Institutes of Health(美国国立卫生研究院国家医学图书馆);University of Virginia(弗吉尼亚大学);University of Illinois at Urbana Champaign(伊利诺伊大学厄巴纳-香槟分校);Lapis Labs(拉皮斯实验室);Yale University School of Medicine(耶鲁大学医学院);Rosalind Franklin University Chicago Medical School(罗莎琳德·富兰克林大学芝加哥医学院);Howard University College of Medicine(霍华德大学医学院);University of Illinois College of Medicine at Chicago(芝加哥伊利诺伊大学医学院);University of Chicago Pritzker School of Medicine(芝加哥大学普利兹克医学院)
MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models
University of Tübingen(蒂宾根大学);Tübingen AI Center(蒂宾根人工智能中心);Helmholtz Munich(亥姆霍兹慕尼黑中心);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);University of Virginia(弗吉尼亚大学);University of British Columbia(不列颠哥伦比亚大学);Harvard Medical School(哈佛医学院);Technical University of Munich(慕尼黑工业大学)
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
East China Normal University(华东师范大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Hasso Plattner Institute(哈索·普拉特纳研究所);University of Potsdam(波茨坦大学);Cooperative Medianet Innovation Center, Shanghai Jiao Tong University(上海交通大学合作媒体网络创新中心)
Evaluating the Impact of Lab Test Results on Large Language Models Generated Differential Diagnoses from Clinical Case Vignettes
Florida State University(佛罗里达州立大学);National Center for Biotechnology Information, National Library of Medicine(美国国家医学图书馆国家生物技术信息中心);Emory University(埃默里大学);Morsani College of Medicine, University of South Florida(南佛罗里达大学莫萨尼医学院)
SUP-MIMIC: A Multi-Task Clinical Diagnosis Benchmark for Evaluating LLMs' Robustness to Contradictory Evidence
Beijing University of Technology(北京工业大学);Beijing Institute of Technology(北京理工大学);China-Japan Friendship Hospital(中日友好医院);Dongbei University of Finance and Economics(东北财经大学)
InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents
Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);Zhejiang University(浙江大学);University of Oxford(牛津大学);MemTensor (Shanghai) Technology Co., Ltd(MemTensor(上海)科技有限公司)
PersianMedQA: Evaluating Large Language Models on a Persian-English Bilingual Medical Question Answering Benchmark
School of Electrical and Computer Engineering, University of Tehran(德黑兰大学电气与计算机工程学院);Shahid Beheshti University of Medical Sciences(沙希德·贝赫什提医学科学大学);Institute for Research in Fundamental Sciences (IPM)(基础科学研究所(IPM))
SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture
Department of Computer Science and Engineering, Indian Institute of Technology Patna(印度理工学院帕纳布分校计算机科学与工程系);Department of Computer Science, Banasthali Vidyapeeth University(班斯塔利大学计算机科学系)
TCM-Ladder: A Benchmark for Multimodal Question Answering on Traditional Chinese Medicine
Department of Electrical Engineering and Computer Science, University of Missouri(密苏里大学电气工程与计算机科学系);Christopher S. Bond Life Sciences Center, University of Missouri(密苏里大学克里斯托弗·S·邦德生命科学中心);Department of Computer Science, Northwestern University(西北大学计算机科学系);Department of Computer Science and Mathematics, Truman State University(特拉华州立大学计算机科学与数学系);Marquette High School(马基特高中);Community Health Service Center, Shanghai Pudong New Area(上海浦东新区社区卫生服务中心);School of Engineering and Applied Science, University of Pennsylvania(宾夕法尼亚大学工程与应用科学学院);Department of Endocrinology, Seventh People’s Hospital of Shanghai University of Traditional Chinese Medicine(上海中医药大学第七人民医院内分泌科);School of Acupuncture-Moxibustion and Tuina, Shanghai University of Traditional Chinese Medicine(上海中医药大学针灸推拿学院)