arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.13833cs.CL
收藏

生物医学大语言模型在未见医学数据上似乎并不优于通用模型

Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data

Charité - Universitätsmedizin Berlin(柏林夏里特医学院);Freie Universität Berlin(柏林自由大学);Humboldt Universität zu Berlin(柏林洪堡大学);Athinoula A. Martinos Center for Biomedical Imaging(雅典娜·马蒂诺斯生物医学成像中心);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);Institute for AI in Medicine (IKIM)(医学人工智能研究所);University Hospital Essen (AöR)(埃森大学医院);Technical University of Munich(慕尼黑工业大学);University Hospital Aachen(亚琛大学医院);Cancer Research Center Cologne Essen (CCCE)(科隆埃森癌症研究中心);West German Cancer Center Essen(西德埃森癌症中心);German Cancer Consortium (DKTK, Partner site Essen)(德国癌症联盟(埃森合作站点));TU Dortmund(多特蒙德工业大学);Bakar Computational Health Sciences Institute(巴卡尔计算健康科学研究所);University of California, San Francisco(加州大学旧金山分校)

Felix J. Dorfner, Amin Dada, Felix Busch, Marcus R. Makowski, Tianyu Han, Daniel Truhn, Jens Kleesiek, Madhumita Sushil, Jacqueline Lammert, Lisa C. Adams, Keno K. Bressem

AI总结:

本研究评估生物医学微调LLMs在未见临床数据上的表现,发现其总体逊于通用模型,表明领域微调可能降低性能,建议采用检索增强生成等替代方法。

摘要:

大语言模型(LLMs)在生物医学应用中展现了潜力,促使人们致力于在领域特定数据上对其进行微调。然而,这种方法的有效性仍不明确。本研究评估了生物医学微调LLMs与通用对应模型在多种临床任务上的表现。我们评估了它们在《新英格兰医学杂志》(NEJM)和《美国医学会杂志》(JAMA)的临床病例挑战以及多项临床任务(例如,信息提取、文档摘要和临床编码)上的表现。使用特意选择可能不在生物医学模型微调数据集内的基准,我们发现生物医学LLMs的表现大多逊于通用对应模型,尤其是在不侧重医学知识的任务上。虽然较大模型在病例任务上表现相似(例如,在JAMA病例上,OpenBioLLM-70B:66.4% vs. Llama-3-70B-Instruct:65%),但较小的生物医学模型表现不佳更为明显(例如,在NEJM病例上,OpenBioLLM-8B:30% vs. Llama-3-8B-Instruct:64.3%)。在CLUE(临床语言理解评估)基准任务中也观察到类似趋势,通用模型在文本生成、问答和编码任务上通常表现更好。我们的结果表明,将LLMs微调至生物医学数据可能无法提供预期收益,甚至可能导致性能下降,这挑战了关于LLMs领域特定适应的普遍假设,并凸显了医疗AI中需要更严格的评估框架。替代方法,如检索增强生成,在增强LLMs生物医学能力且不损害其通用知识方面可能更为有效。

英文摘要:

Large language models (LLMs) have shown potential in biomedical applications, leading to efforts to fine-tune them on domain-specific data. However, the effectiveness of this approach remains unclear. This study evaluates the performance of biomedically fine-tuned LLMs against their general-purpose counterparts on a variety of clinical tasks. We evaluated their performance on clinical case challenges from the New England Journal of Medicine (NEJM) and the Journal of the American Medical Association (JAMA) and on several clinical tasks (e.g., information extraction, document summarization, and clinical coding). Using benchmarks specifically chosen to be likely outside the fine-tuning datasets of biomedical models, we found that biomedical LLMs mostly perform inferior to their general-purpose counterparts, especially on tasks not focused on medical knowledge. While larger models showed similar performance on case tasks (e.g., OpenBioLLM-70B: 66.4% vs. Llama-3-70B-Instruct: 65% on JAMA cases), smaller biomedical models showed more pronounced underperformance (e.g., OpenBioLLM-8B: 30% vs. Llama-3-8B-Instruct: 64.3% on NEJM cases). Similar trends were observed across the CLUE (Clinical Language Understanding Evaluation) benchmark tasks, with general-purpose models often performing better on text generation, question answering, and coding tasks. Our results suggest that fine-tuning LLMs to biomedical data may not provide the expected benefits and may potentially lead to reduced performance, challenging prevailing assumptions about domain-specific adaptation of LLMs and highlighting the need for more rigorous evaluation frameworks in healthcare AI. Alternative approaches, such as retrieval-augmented generation, may be more effective in enhancing the biomedical capabilities of LLMs without compromising their general knowledge.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑