arXiv 2304.144542023-08-25cs.CL

PMC-LLaMA：致力于构建医疗领域开源语言模型

PMC-LLaMA: Towards Building Open-source Language Models for Medicine

Cooperative Medianet Innovation Center（合作中位数创新中心）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）

Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, Weidi Xie

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对通用大语言模型医疗领域知识不足的问题，研究团队通过注入海量医学生物文献数据、构建大规模指令微调数据集并微调，打造了开源医疗模型PMC-LLaMA，130亿参数规模的轻量版在医学问答基准上性能超越ChatGPT。

摘要：

近年来，大语言模型（LLMs）在自然语言理解方面展现出了卓越能力。尽管这类模型在日常对话和问答场景中表现出色，但由于缺乏领域专属知识，它们在医疗等对精度有要求的领域往往表现不佳。本文介绍了一款专为医疗应用打造的强大开源语言模型的构建流程，该模型名为PMC-LLaMA。我们的贡献分为三方面：（1）系统研究了将通用基础语言模型适配医疗领域的过程，包括通过整合480万篇生物医学学术论文和3万本医学教科书开展以数据为中心的知识注入，以及为适配领域专属指令进行的全面微调；（2）贡献了一个大规模、综合性的指令微调数据集，该数据集涵盖医学问答（QA）、推理解释以及对话内容，总token量达2.02亿；（3）开展了全面的消融实验，验证了所提出各组件的有效性。在多个公开医学问答基准上进行评估时，我们仅含130亿参数的轻量型PMC-LLaMA表现优异，甚至超越了ChatGPT。所有模型、代码、数据集均可在https://github.com/chaoyi-wu/PMC-LLaMA 获取。

英文摘要：

Recently, Large Language Models (LLMs) have showcased remarkable capabilities in natural language understanding. While demonstrating proficiency in everyday conversations and question-answering situations, these models frequently struggle in domains that require precision, such as medical applications, due to their lack of domain-specific knowledge. In this paper, we describe the procedure for building a powerful, open-source language model specifically designed for medicine applications, termed as PMC-LLaMA. Our contributions are threefold: (i) we systematically investigate the process of adapting a general-purpose foundation language model towards medical domain, this involves data-centric knowledge injection through the integration of 4.8M biomedical academic papers and 30K medical textbooks, as well as comprehensive fine-tuning for alignment with domain-specific instructions; (ii) we contribute a large-scale, comprehensive dataset for instruction tuning. This dataset encompasses medical question-answering (QA), rationale for reasoning, and conversational dialogues, comprising a total of 202M tokens; (iii) we conduct thorough ablation studies to demonstrate the effectiveness of each proposed component. While evaluating on various public medical question-answering benchmarks, our lightweight PMCLLaMA, which consists of only 13 billion parameters, exhibits superior performance, even surpassing ChatGPT. All models, codes, datasets can be found in https://github.com/chaoyi-wu/PMC-LLaMA.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。