LLMs have become increasingly capable at accomplishing a range of specialized-tasks and can be utilized to expand equitable access to medical knowledge. Most medical LLMs have involved extensive fine-tuning, leveraging specialized medical data and significant, thus costly, amounts of computational power. Many of the top performing LLMs are proprietary and their access is limited to very few research groups. However, open-source (OS) models represent a key area of growth for medical LLMs due to significant improvements in performance and an inherent ability to provide the transparency and compliance required in healthcare. We present OpenMedLM, a prompting platform which delivers state-of-the-art (SOTA) performance for OS LLMs on medical benchmarks. We evaluated a range of OS foundation LLMs (7B-70B) on four medical benchmarks (MedQA, MedMCQA, PubMedQA, MMLU medical-subset). We employed a series of prompting strategies, including zero-shot, few-shot, chain-of-thought (random selection and kNN selection), and ensemble/self-consistency voting. We found that OpenMedLM delivers OS SOTA results on three common medical LLM benchmarks, surpassing the previous best performing OS models that leveraged computationally costly extensive fine-tuning. The model delivers a 72.6% accuracy on the MedQA benchmark, outperforming the previous SOTA by 2.4%, and achieves 81.7% accuracy on the MMLU medical-subset, establishing itself as the first OS LLM to surpass 80% accuracy on this benchmark. Our results highlight medical-specific emergent properties in OS LLMs which have not yet been documented to date elsewhere, and showcase the benefits of further leveraging prompt engineering to improve the performance of accessible LLMs for medical applications.
MediQAl: A French Medical Question Answering Dataset for Knowledge and Reasoning Evaluation
Data Clinic, University Hospital of Nantes, France(南特大学医院数据诊所,法国);Nantes Université, École Centrale Nantes, CNRS, LS2N, France(南特大学,中央理工学院南特分校,国家科学研究中心,LS2N,法国)
PerMedCQA: Benchmarking Large Language Models on Medical Consumer Question Answering in Persian Language
School of Computer Science Institute for Research in Fundamental Sciences(计算机科学学院基础科学研究研究所);School of Electrical and Computer Engineering University of Tehran(电气与计算机工程学院塔里班大学);Department of Computer Science Faculty of Mathematical Sciences Alzahra University(计算机科学系数学科学学院阿扎赫拉大学)
SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering
The Hong Kong Polytechnic University(香港理工大学);West China Hospital of Sichuan University(四川大学华西医院);Sichuan Academy of Medical Sciences(四川省医学科学院);Sichuan Provincial People's Hospital(四川省人民医院)
A Systematic Study of Retrieval Pipeline Design for Retrieval-Augmented Medical Question Answering
Department of Mechatronics & Industrial Engineering, Chittagong University of Engineering & Technology(吉大港工程与技术大学机电与工业工程系);Department of Mechanical Engineering, Chittagong University of Engineering & Technology(吉大港工程与技术大学机械工程系)
RGAR: Recurrence Generation-augmented Retrieval for Factual-aware Medical Question Answering
King’s College London(伦敦国王学院);School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院);School of Electrical and Computer Engineering, Carnegie Mellon University(卡内基梅隆大学电气与计算机工程学院)
Finding Similar Medical Questions from Question Answering Websites
Tencent Medical AI Lab(腾讯医疗人工智能实验室);State University of New York at Buffalo(纽约州立大学布法罗分校);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);JD Urban Computing Business Unit(京东城市计算事业部);University of Illinois at Chicago(伊利诺伊大学芝加哥分校)
Exploring the Role of Knowledge Graph-Based RAG in Japanese Medical Question Answering with Small-Scale LLMs
The University of Tokyo(东京大学);NEC Laboratories America(NEC美国实验室);Dokkyo Medical University(多摩医学大学);Graduate School of Frontier Sciences, The University of Tokyo(东京大学前沿科学研究生院)
CURE: Confidence-driven Unified Reasoning Ensemble Framework for Medical Question Answering
School of Information Technology and Computer Science, Nile University(信息科技与计算机科学学院,尼罗大学);Graduate School of Information Science, University of Hyogo(信息科学研究生院,播磨大学);Advanced Medical Engineering Research Institute, University of Hyogo(先进医疗工程研究所,播磨大学)
MF2-MVQA: A Multi-stage Feature Fusion method for Medical Visual Question Answering
University of Science and Technology Beijing(北京科技大学);Key Laboratory of Knowledge Automation for Industrial Processes, Ministry of Education(教育部工业过程知识自动化重点实验室);Shunde Innovation School, University of Science and Technology Beijing(北京科技大学顺德创新学院)
Optimizing Medical Question-Answering Systems: A Comparative Study of Fine-Tuned and Zero-Shot Large Language Models with RAG Framework
Department of Electrical Engineering;Computer Science University of Toledo Toledo, USA;Institute of Mathematical Sciences Claremont Graduate University Claremont, USA;Department of Bioengineering University of Toledo Toledo, USA;Department of Computer Science Bowling Green State University Bowling Green, USA
Improving Reliability and Explainability of Medical Question Answering through Atomic Fact Checking in Retrieval-Augmented LLMs
Technical University of Munich(慕尼黑工业大学);TUM University Hospital Rechts der Isar(慕尼黑工业大学 Rechts der Isar 大学医院);German Heart Center Munich(慕尼黑德国心脏中心);Helmholtz Zentrum München(慕尼黑亥姆霍兹中心);German Consortium for Translational Cancer Research(德国转化癌症研究联盟)
Integrating UMLS Knowledge into Large Language Models for Medical Question Answering
National University of Singapore(新加坡国立大学);Duke-NUS Medical School(杜克-新加坡国立大学医学院);University of Tokyo(东京大学);National Institute of Advanced Industrial Science and Technology(日本产业技术综合研究所);Singapore General Hospital(新加坡中央医院);Zhejiang Lab(之江实验室);National Institutes of Health(美国国立卫生研究院);Yale University(耶鲁大学)