arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.16079cs.CLcs.AIcs.LG
收藏

MEDITRON-70B:扩展大型语言模型的医学预训练

MEDITRON-70B: Scaling Medical Pretraining for Large Language Models

EPFL(洛桑联邦理工学院);Open Assistant(开放助手);Yale(耶鲁大学);Idiap Research Institute(伊迪亚普研究所)
Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan M… 展开作者

Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, Antoine Bosselut

AI总结:

本研究发布了开源医学LLM套件MEDITRON(7B/70B),基于Llama-2在精选医学语料库上扩展预训练,性能显著优于同类开源基线并逼近GPT-4等闭源模型。

摘要:

大型语言模型(LLMs)有望实现医学知识的普及化。尽管已有许多努力致力于利用和提升LLMs的医学知识与推理能力,但由此产生的模型要么是闭源的(例如PaLM、GPT-4),要么在规模上受限(≤13B参数),这限制了它们的能力。在本研究中,我们通过发布MEDITRON来改善对大规模医学LLMs的访问:这是一套包含7B和70B参数的开源LLMs,已适应医学领域。MEDITRON基于Llama-2构建(通过对Nvidia的Megatron-LM分布式训练器的适配),并在一个精心整理的医学语料库上扩展预训练,该语料库包括精选的PubMed文章、摘要和国际认可的医学指南。使用四个主要医学基准的评估显示,在任务特定微调前后,相对于多个最先进的基线模型均取得了显著的性能提升。总体而言,MEDITRON在其参数类别中相对于最佳公开基线实现了6%的绝对性能提升,相对于我们从Llama-2微调的最强基线实现了3%的提升。与闭源LLMs相比,MEDITRON-70B优于GPT-3.5和Med-PaLM,且与GPT-4的差距在5%以内,与Med-PaLM-2的差距在10%以内。我们发布了用于整理医学预训练语料库的代码以及MEDITRON模型权重,以推动更强大医学LLMs的开源发展。

英文摘要:

Large language models (LLMs) can potentially democratize access to medical knowledge. While many efforts have been made to harness and improve LLMs' medical knowledge and reasoning capacities, the resulting models are either closed-source (e.g., PaLM, GPT-4) or limited in scale (<= 13B parameters), which restricts their abilities. In this work, we improve access to large-scale medical LLMs by releasing MEDITRON: a suite of open-source LLMs with 7B and 70B parameters adapted to the medical domain. MEDITRON builds on Llama-2 (through our adaptation of Nvidia's Megatron-LM distributed trainer), and extends pretraining on a comprehensively curated medical corpus, including selected PubMed articles, abstracts, and internationally-recognized medical guidelines. Evaluations using four major medical benchmarks show significant performance gains over several state-of-the-art baselines before and after task-specific finetuning. Overall, MEDITRON achieves a 6% absolute performance gain over the best public baseline in its parameter class and 3% over the strongest baseline we finetuned from Llama-2. Compared to closed-source LLMs, MEDITRON-70B outperforms GPT-3.5 and Med-PaLM and is within 5% of GPT-4 and 10% of Med-PaLM-2. We release our code for curating the medical pretraining corpus and the MEDITRON model weights to drive open-source development of more capable medical LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑