arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.00588cs.CLcs.AI
收藏

缩小开源与商用大型语言模型在医学证据摘要方面的差距

Closing the gap between open-source and commercial large language models for medical evidence summarization

Columbia University(哥伦比亚大学);National Library of Medicine(美国国家医学图书馆);National Institutes of Health(美国国立卫生研究院);Weill Cornell Medicine(威尔康奈尔医学院);The University of Texas at Austin(德克萨斯大学奥斯汀分校);Icahn School of Medicine at Mount Sinai(西奈山伊坎医学院)
Gongbo Zhang, Qiao Jin, Yiliang Zhou, Song Wang, Betina R. Idnay, Yiming Luo, Elizabeth Park, Jordan G. Nestor, Matthew E. Spotnitz, Ali Soroush, Thomas Campion… 展开作者

Gongbo Zhang, Qiao Jin, Yiliang Zhou, Song Wang, Betina R. Idnay, Yiming Luo, Elizabeth Park, Jordan G. Nestor, Matthew E. Spotnitz, Ali Soroush, Thomas Campion, Zhiyong Lu, Chunhua Weng, Yifan Peng

AI总结:

本研究通过微调PRIMERA、LongT5和Llama-2三种开源LLMs,在MedReview数据集上显著提升医学证据摘要性能,缩小了与商用模型的差距,并为领域特定任务的模型选择提供依据。

摘要:

大型语言模型(LLMs)在总结医学证据方面展现出巨大潜力。近期研究大多聚焦于专有LLMs的应用。使用专有LLMs会引入多种风险因素,包括缺乏透明度和供应商依赖。尽管开源LLMs能够提供更好的透明度和可定制性,但其性能与专有模型相比仍有差距。在本研究中,我们探究了微调开源LLMs能在多大程度上进一步提升其在医学证据摘要中的性能。利用包含8,161对系统综述与摘要的基准数据集MedReview,我们对三种广泛使用的开源LLMs——即PRIMERA、LongT5和Llama-2——进行了微调。总体而言,微调后的LLMs在ROUGE-L上提升了9.89(95%置信区间:8.94-10.81),在METEOR分数上提升了13.21(95%置信区间:12.05-14.37),在CHRF分数上提升了15.82(95%置信区间:13.89-16.44)。微调后的LongT5性能接近零样本设置下的GPT-3.5。此外,较小的微调模型有时甚至表现出优于较大的零样本模型的性能。上述改进趋势在人工评估和GPT4模拟评估中均得到体现。我们的结果可用于指导对需要特定领域知识(如医学证据摘要)的任务进行模型选择。

英文摘要:

Large language models (LLMs) hold great promise in summarizing medical evidence. Most recent studies focus on the application of proprietary LLMs. Using proprietary LLMs introduces multiple risk factors, including a lack of transparency and vendor dependency. While open-source LLMs allow better transparency and customization, their performance falls short compared to proprietary ones. In this study, we investigated to what extent fine-tuning open-source LLMs can further improve their performance in summarizing medical evidence. Utilizing a benchmark dataset, MedReview, consisting of 8,161 pairs of systematic reviews and summaries, we fine-tuned three broadly-used, open-sourced LLMs, namely PRIMERA, LongT5, and Llama-2. Overall, the fine-tuned LLMs obtained an increase of 9.89 in ROUGE-L (95% confidence interval: 8.94-10.81), 13.21 in METEOR score (95% confidence interval: 12.05-14.37), and 15.82 in CHRF score (95% confidence interval: 13.89-16.44). The performance of fine-tuned LongT5 is close to GPT-3.5 with zero-shot settings. Furthermore, smaller fine-tuned models sometimes even demonstrated superior performance compared to larger zero-shot models. The above trends of improvement were also manifested in both human and GPT4-simulated evaluations. Our results can be applied to guide model selection for tasks demanding particular domain knowledge, such as medical evidence summarization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑