Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data
Charité - Universitätsmedizin Berlin(柏林夏里特医学院);Freie Universität Berlin(柏林自由大学);Humboldt Universität zu Berlin(柏林洪堡大学);Athinoula A. Martinos Center for Biomedical Imaging(雅典娜·马蒂诺斯生物医学成像中心);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);Institute for AI in Medicine (IKIM)(医学人工智能研究所);University Hospital Essen (AöR)(埃森大学医院);Technical University of Munich(慕尼黑工业大学);University Hospital Aachen(亚琛大学医院);Cancer Research Center Cologne Essen (CCCE)(科隆埃森癌症研究中心);West German Cancer Center Essen(西德埃森癌症中心);German Cancer Consortium (DKTK, Partner site Essen)(德国癌症联盟(埃森合作站点));TU Dortmund(多特蒙德工业大学);Bakar Computational Health Sciences Institute(巴卡尔计算健康科学研究所);University of California, San Francisco(加州大学旧金山分校)
Felix J. Dorfner, Amin Dada, Felix Busch, Marcus R. Makowski, Tianyu Han, Daniel Truhn, Jens Kleesiek, Madhumita Sushil, Jacqueline Lammert, Lisa C. Adams, Keno K. Bressem
大语言模型(LLMs)在生物医学应用中展现了潜力,促使人们致力于在领域特定数据上对其进行微调。然而,这种方法的有效性仍不明确。本研究评估了生物医学微调LLMs与通用对应模型在多种临床任务上的表现。我们评估了它们在《新英格兰医学杂志》(NEJM)和《美国医学会杂志》(JAMA)的临床病例挑战以及多项临床任务(例如,信息提取、文档摘要和临床编码)上的表现。使用特意选择可能不在生物医学模型微调数据集内的基准,我们发现生物医学LLMs的表现大多逊于通用对应模型,尤其是在不侧重医学知识的任务上。虽然较大模型在病例任务上表现相似(例如,在JAMA病例上,OpenBioLLM-70B:66.4% vs. Llama-3-70B-Instruct:65%),但较小的生物医学模型表现不佳更为明显(例如,在NEJM病例上,OpenBioLLM-8B:30% vs. Llama-3-8B-Instruct:64.3%)。在CLUE(临床语言理解评估)基准任务中也观察到类似趋势,通用模型在文本生成、问答和编码任务上通常表现更好。我们的结果表明,将LLMs微调至生物医学数据可能无法提供预期收益,甚至可能导致性能下降,这挑战了关于LLMs领域特定适应的普遍假设,并凸显了医疗AI中需要更严格的评估框架。替代方法,如检索增强生成,在增强LLMs生物医学能力且不损害其通用知识方面可能更为有效。
英文摘要:
Large language models (LLMs) have shown potential in biomedical applications, leading to efforts to fine-tune them on domain-specific data. However, the effectiveness of this approach remains unclear. This study evaluates the performance of biomedically fine-tuned LLMs against their general-purpose counterparts on a variety of clinical tasks. We evaluated their performance on clinical case challenges from the New England Journal of Medicine (NEJM) and the Journal of the American Medical Association (JAMA) and on several clinical tasks (e.g., information extraction, document summarization, and clinical coding). Using benchmarks specifically chosen to be likely outside the fine-tuning datasets of biomedical models, we found that biomedical LLMs mostly perform inferior to their general-purpose counterparts, especially on tasks not focused on medical knowledge. While larger models showed similar performance on case tasks (e.g., OpenBioLLM-70B: 66.4% vs. Llama-3-70B-Instruct: 65% on JAMA cases), smaller biomedical models showed more pronounced underperformance (e.g., OpenBioLLM-8B: 30% vs. Llama-3-8B-Instruct: 64.3% on NEJM cases). Similar trends were observed across the CLUE (Clinical Language Understanding Evaluation) benchmark tasks, with general-purpose models often performing better on text generation, question answering, and coding tasks. Our results suggest that fine-tuning LLMs to biomedical data may not provide the expected benefits and may potentially lead to reduced performance, challenging prevailing assumptions about domain-specific adaptation of LLMs and highlighting the need for more rigorous evaluation frameworks in healthcare AI. Alternative approaches, such as retrieval-augmented generation, may be more effective in enhancing the biomedical capabilities of LLMs without compromising their general knowledge.
GLiNER-BioMed: A Suite of Efficient Models for Open Biomedical Named Entity Recognition
Department of Radiology and Medical Informatics, Faculty of Medicine, University of Geneva(日内瓦大学医学院放射学与医学信息学系);Knowledgator Engineering(Knowledgator工程)
BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
Lehigh University(理海大学);University of Georgia(佐治亚大学);Samsung Research America(三星美国研究院);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);University of Pennsylvania(宾夕法尼亚大学);Children’s Hospital of Philadelphia(费城儿童医院);University of Central Florida(中佛罗里达大学);University of California, Santa Cruz(加州大学圣克鲁兹分校);UTHealth Houston(休斯顿德克萨斯大学健康科学中心);Mayo Clinic(妙佑医疗国际);Penn Institute for Biomedical Informatics(宾夕法尼亚生物医学信息学研究所);Leonard Davis Institute of Health Economics(伦纳德·戴维斯健康经济学研究所);Stanford University School of Medicine(斯坦福大学医学院);Stanford University(斯坦福大学)
A Unified Biomedical Named Entity Recognition Framework with Large Language Models
College of Computer Science and Technology, Dalian University of Technology(大连理工大学计算机科学与技术学院);Air Force Communications NCO Academy(空军通信NCO学院);Department of Pharmacy, Second Affiliated Hospital of Dalian Medical University(大连医科大学第二附属医院药学部)
UltraMedical: Building Specialized Generalists in Biomedicine
Tsinghua University(清华大学);University of Washington(华盛顿大学);The First Affiliated Hospital of Nanchang University(南昌大学第一附属医院);Shanghai Jiao Tong University(上海交通大学)
Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation
Tsinghua University(清华大学);Shanghai AI Laboratory(上海人工智能实验室);Harbin Institute of Technology(哈尔滨工业大学);University of Washington(华盛顿大学);The First Affiliated Hospital of Nanchang University(南昌大学第一附属医院)
TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification
Mohamed bin Zayed University of Artificial Intelligence(莫莫德·本·扎耶德人工智能大学);Monash University(墨尔本大学);Xi’an Jiaotong-Liverpool University(西安交通大学利物浦大学);University of Southampton(南安普顿大学)
GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究所);University of Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国立大学)
MUDI: A Multimodal Biomedical Dataset for Understanding Pharmacodynamic Drug-Drug Interactions
VNU University of Engineering and Technology (VNU-UET)(越南工程技术大学);Lausanne University Hospital (CHUV)(洛桑大学医院);University of Lausanne (UNIL)(洛桑大学);Hanoi Medical University(河内医学院);National Geriatric Hospital, Hanoi(河内国立老年医院)