Med42 -- Evaluating Fine-Tuning Strategies for Medical LLMs: Full-Parameter vs. Parameter-Efficient Approaches
G42 Healthcare(G42 医疗健康);Core42;Cerebras Systems
Clément Christophe, Praveen K Kanithi, Prateek Munjal, Tathagata Raha, Nasir Hayat, Ronnie Rajan, Ahmed Al-Mahrooqi, Avani Gupta, Muhammad Umar Salman, Gurpreet… 展开作者
Clément Christophe, Praveen K Kanithi, Prateek Munjal, Tathagata Raha, Nasir Hayat, Ronnie Rajan, Ahmed Al-Mahrooqi, Avani Gupta, Muhammad Umar Salman, Gurpreet Gosal, Bhargav Kanakiya, Charles Chen, Natalia Vassilieva, Boulbaba Ben Amor, Marco AF Pimentel, Shadab Khan
备注:
Published at AAAI 2024 Spring Symposium - Clinical Foundation Models
This study presents a comprehensive analysis and comparison of two predominant fine-tuning methodologies - full-parameter fine-tuning and parameter-efficient tuning - within the context of medical Large Language Models (LLMs). We developed and refined a series of LLMs, based on the Llama-2 architecture, specifically designed to enhance medical knowledge retrieval, reasoning, and question-answering capabilities. Our experiments systematically evaluate the effectiveness of these tuning strategies across various well-known medical benchmarks. Notably, our medical LLM Med42 showed an accuracy level of 72% on the US Medical Licensing Examination (USMLE) datasets, setting a new standard in performance for openly available medical LLMs. Through this comparative analysis, we aim to identify the most effective and efficient method for fine-tuning LLMs in the medical domain, thereby contributing significantly to the advancement of AI-driven healthcare applications.
MIRIAD: Augmenting LLMs with millions of medical query-response pairs
Department of Biosystems Science and Engineering, ETH Zurich(生物系统科学与工程系,苏黎世联邦理工学院);Department of Computer Science, Stanford University(计算机科学系,斯坦福大学);Department of Internal Medicine, Mayo Clinic(内科医学系,梅奥诊所);Hugging Face;Department of Radiology, Stanford University(放射学系,斯坦福大学);Hasso-Plattner-Institute for Digital Engineering, University of Potsdam(数字工程研究所,波茨坦大学);Center for Artificial Intelligence in Medicine and Imaging, Stanford, CA, USA(医学与成像人工智能中心,斯坦福,CA,美国);Scripps Translational Science Institute, San Diego, CA, USA(斯克里普斯转化科学研究所,圣地亚哥,CA,美国)
HiMed: Incentivizing Hindi Reasoning in Medical LLMs
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Indian Institute of Technology (Banaras Hindu University) Varanasi(印度理工学院(班加罗尔 Hindu 大学)瓦拉纳西分校);Tongji University(同济大学);Shenzhen Research Institute of Big Data(深圳大数据研究院);Shenzhen Loop Area Institute(深圳河套学院);The Hong Kong University of Science and Technology(香港科技大学);Halmstad University(哈尔姆斯塔德大学)
Performance of large language models in numerical vs. semantic medical knowledge: Benchmarking on evidence-based Q&As
Tel-Aviv University(特拉维夫大学);Kahun Medical Ltd(Kahun医疗有限公司);Hebrew University of Jerusalem(耶路撒冷希伯来大学);Leumit Health Services(Leumit医疗服务机构);Bar-Ilan University(巴伊兰大学);Kaplan Medical Center(卡普兰医疗中心);Brigham and Women’s Hospital(布莱根妇女医院);Harvard Medical School(哈佛医学院)
Enabling Doctor-Centric Medical AI with LLMs through Workflow-Aligned Tasks and Benchmarks
Shenzhen Research Institute of Big Data(大数据研究 institute);National Health Data Institute(国家健康数据研究所);Halmstad University(哈马碧大学);Shenzhen University of Advanced Technology(深圳先进技术大学)
Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence
The University of Texas at Austin(德克萨斯大学奥斯汀分校);Northeastern University(东北大学);MD Anderson Cancer Center(MD安德森癌症中心);Georgia Institute of Technology(佐治亚理工学院)
The Biased Oracle: Assessing LLMs' Understandability and Empathy in Medical Diagnoses
Department of Chemistry and Applied Biosciences, ETH Zurich(应用生物科学系,苏黎世联邦理工学院);Department of Computer Science, ETH Zurich(计算机科学系,苏黎世联邦理工学院);Department of Health Sciences and Technology, ETH Zurich(健康科学与技术系,苏黎世联邦理工学院)
A Super-Learner with Large Language Models for Medical Emergency Advising
Northeastern University(东北大学);Biovitalage S.R.L.(Biovitalage有限责任公司);Territorial Emergency System SET 118(地区紧急系统SET 118);Territorial Center for Medical Assistance(地区医疗援助中心);Aldo Moro University of Bari(巴里阿尔多·莫罗大学)
Evaluating large language models in medical applications: a survey
The Hong Kong Polytechnic University(香港理工大学);Research Centre for SHARP Vision (RCSV), The Hong Kong Polytechnic University(香港理工大学SHARP视觉研究中心);Centre for Eye and Vision Research (CEVR)(眼与视觉研究中心);Renji Hospital, School of Medicine, Shanghai Jiao Tong University(上海交通大学医学院附属仁济医院);Perception Vision Medical Technologies Co. Ltd.(感知视觉医疗科技有限公司);The Hong Kong University of Science and Technology(香港科技大学)
Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving
Department of Electronic Engineering, Tsinghua University, Beijing, China(清华大学电子工程系);College of AI, Tsinghua University, Beijing, China(清华大学人工智能学院);School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China(北京邮电大学计算机科学学院);iFLYTEK Research, Hefei, China(iFLYTEK研究院)
A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations
School of Computer & Communication Engineering, University of Science and Technology Beijing(北京科技大学计算机与通信工程学院);School of Computer Science, University of South China(南华大学计算机学院);Faculty of Computer and Information Sciences, Hosei University(法政大学计算机与信息科学学院)
Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgments
University of Georgia(佐治亚大学);University of California, Merced(加州大学默塞德分校);Lehigh University(莱特大学);The University of Queensland(昆士兰大学);National University of Singapore(新加坡国立大学);University of California, Los Angeles(加州大学洛杉矶分校)
JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Models
Kyoto University(京都大学);Hohai University(河海大学);The University of Tokyo(东京大学);University of Science and Technology of China(中国科学技术大学);Hong Kong Polytechnic University(香港理工大学)
Structured Outputs Enable General-Purpose LLMs to be Medical Experts
University of Bristol(布里斯托大学);Lehigh University(利哈伊大学);The University of Queensland(昆士兰大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of California, Merced(加利福尼亚大学默塞德分校)
MedCalc-Bench: Evaluating Large Language Models for Medical Calculations
National Library of Medicine, National Institutes of Health(美国国立卫生研究院国家医学图书馆);University of Virginia(弗吉尼亚大学);University of Illinois at Urbana Champaign(伊利诺伊大学厄巴纳-香槟分校);Lapis Labs(拉皮斯实验室);Yale University School of Medicine(耶鲁大学医学院);Rosalind Franklin University Chicago Medical School(罗莎琳德·富兰克林大学芝加哥医学院);Howard University College of Medicine(霍华德大学医学院);University of Illinois College of Medicine at Chicago(芝加哥伊利诺伊大学医学院);University of Chicago Pritzker School of Medicine(芝加哥大学普利兹克医学院)
The Potential of LLMs in Medical Education: Generating Questions and Answers for Qualification Exams
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Beijing Friendship Hospital, Capital Medical University(首都医科大学附属北京友谊医院);Xunfei Healthcare Technology Co., Ltd(讯飞医疗科技有限公司);School of Information and Communication Engineering, Hainan University(海南大学信息与通信工程学院);Guangzhou University(广州大学)