arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.19371cs.CLcs.AIcs.IR
收藏

OpenMedLM:在开源大语言模型的医学问答任务中,提示工程的表现可优于微调

OpenMedLM: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models

Montera, Inc dba Forta(Montera公司(经营名称为Forta))

Jenish Maharjan, Anurag Garikipati, Navan Preet Singh, Leo Cyrus, Mayank Sharma, Madalina Ciobanu, Gina Barnes, Rahul Thapa, Qingqing Mao, Ritankar Das

AI总结:

本研究提出开源医学提示平台OpenMedLM,通过组合多种提示策略,在多个医学基准上超越依赖高成本微调的开源SOTA模型,证明提示工程可有效提升开源大模型的医学应用性能。

摘要:

大语言模型(LLMs)在完成各类专业任务方面的能力日益增强,可用于扩大医疗知识的公平获取渠道。大多数医学大语言模型都需要利用专业医学数据进行大量微调,这需要消耗大量且成本高昂的计算资源。许多性能顶尖的大语言模型属于专有模型,仅有极少数研究团队能够使用。然而,开源(OS)模型凭借性能的显著提升,以及具备医疗保健领域所需的透明度和合规性的固有优势,成为医学大语言模型发展的关键领域。我们提出了OpenMedLM,这是一个提示工程平台,能让开源大语言模型在医学基准测试中达到当前最优(SOTA)性能。我们在四个医学基准测试集(MedQA、MedMCQA、PubMedQA、MMLU医学子集)上评估了一系列参数规模为70亿到700亿的开源基础大语言模型。我们采用了多种提示策略,包括零样本、少样本、思维链(随机选择和k近邻选择)以及集成/自一致性投票。研究发现,OpenMedLM在三个常见的医学大语言模型基准测试中取得了开源当前最优结果,超越了此前依赖计算成本高昂的大量微调实现最佳性能的开源模型。该模型在MedQA基准测试上的准确率达到72.6%,较此前的当前最优水平高出2.4%;在MMLU医学子集上的准确率达到81.7%,成为首个在该基准测试上准确率突破80%的开源大语言模型。我们的研究结果揭示了开源大语言模型中尚未在其他地方被记录的医学特定涌现特性,也展现了进一步利用提示工程提升可获取大语言模型在医疗应用中性能的优势。

英文摘要:

LLMs have become increasingly capable at accomplishing a range of specialized-tasks and can be utilized to expand equitable access to medical knowledge. Most medical LLMs have involved extensive fine-tuning, leveraging specialized medical data and significant, thus costly, amounts of computational power. Many of the top performing LLMs are proprietary and their access is limited to very few research groups. However, open-source (OS) models represent a key area of growth for medical LLMs due to significant improvements in performance and an inherent ability to provide the transparency and compliance required in healthcare. We present OpenMedLM, a prompting platform which delivers state-of-the-art (SOTA) performance for OS LLMs on medical benchmarks. We evaluated a range of OS foundation LLMs (7B-70B) on four medical benchmarks (MedQA, MedMCQA, PubMedQA, MMLU medical-subset). We employed a series of prompting strategies, including zero-shot, few-shot, chain-of-thought (random selection and kNN selection), and ensemble/self-consistency voting. We found that OpenMedLM delivers OS SOTA results on three common medical LLM benchmarks, surpassing the previous best performing OS models that leveraged computationally costly extensive fine-tuning. The model delivers a 72.6% accuracy on the MedQA benchmark, outperforming the previous SOTA by 2.4%, and achieves 81.7% accuracy on the MMLU medical-subset, establishing itself as the first OS LLM to surpass 80% accuracy on this benchmark. Our results highlight medical-specific emergent properties in OS LLMs which have not yet been documented to date elsewhere, and showcase the benefits of further leveraging prompt engineering to improve the performance of accessible LLMs for medical applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑