arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.01040cs.CL
收藏

从新手到专家:将医学知识建模融入通用大语言模型

From Beginner to Expert: Modeling Medical Knowledge into General LLMs

Ant Group(蚂蚁集团);The Chinese University of Hong Kong, Shenzhen (CUHK-SZ)(香港中文大学(深圳))
Qiang Li, Xiaoyan Yang, Haowen Wang, Qin Wang, Lei Liu, Junjie Wang, Yang Zhang, Mingyuan Chu, Sen Hu, Yicheng Chen, Yue Shen, Cong Fan, Wangshu Zhang, Teng Xu,… 展开作者

Qiang Li, Xiaoyan Yang, Haowen Wang, Qin Wang, Lei Liu, Junjie Wang, Yang Zhang, Mingyuan Chu, Sen Hu, Yicheng Chen, Yue Shen, Cong Fan, Wangshu Zhang, Teng Xu, Jinjie Gu, Jing Zheng, Guannan Zhang Ant Group

AI总结:

本文提出三阶段优化方法将通用大模型微调为医学专家模型AntGLM-Med-10B,构建多类型医学数据集,并创新Verification-of-Choice提示工程,在PubMedQA上超越多数更大规模的模型。

摘要:

最近,基于大语言模型(LLM)的人工智能(AI)系统在自然语言理解和生成方面展现出了卓越的能力。然而,当这些模型面临敏感应用时,例如以类似医生的方式对医学知识进行推理和回答医学问题,它们面临着巨大的挑战。先前的研究试图通过增加模型规模(>100B)来学习更通用的医学知识以克服这一挑战,但在较小规模模型(<100B)的LLM中仍有改进空间。在这项工作中,我们从一个预训练的通用LLM模型(AntGLM-10B)开始,将其从医学新手微调为医学专家(称为AntGLM-Med-10B),该过程利用了三阶段优化程序,即通用医学知识注入、医学领域指令微调和特定医学任务适应。我们的贡献有三个方面:(1)我们专门研究如何使预训练的通用LLM适应医学领域,特别是针对特定的医学任务。(2)我们收集并构建了大规模医学数据集,用于优化过程的每个阶段。这些数据集包含各种数据类型和任务,如问答、医学推理、多选题和医学对话。(3)专门针对医学领域的多选题,我们提出了一种新颖的Verification-of-Choice方法用于提示工程,显著增强了LLM的推理能力。值得注意的是,通过结合上述方法,我们的AntGLM-Med-10B模型在PubMedQA上的表现优于大多数LLM,包括通用和医学LLM,即使这些LLM具有更大的模型规模。

英文摘要:

Recently, large language model (LLM) based artificial intelligence (AI) systems have demonstrated remarkable capabilities in natural language understanding and generation. However, these models face a significant challenge when it comes to sensitive applications, such as reasoning over medical knowledge and answering medical questions in a physician-like manner. Prior studies attempted to overcome this challenge by increasing the model size (>100B) to learn more general medical knowledge, while there is still room for improvement in LLMs with smaller-scale model sizes (<100B). In this work, we start from a pre-trained general LLM model (AntGLM-10B) and fine-tune it from a medical beginner towards a medical expert (called AntGLM-Med-10B), which leverages a 3-stage optimization procedure, i.e., general medical knowledge injection, medical domain instruction tuning, and specific medical task adaptation. Our contributions are threefold: (1) We specifically investigate how to adapt a pre-trained general LLM in medical domain, especially for a specific medical task. (2) We collect and construct large-scale medical datasets for each stage of the optimization process. These datasets encompass various data types and tasks, such as question-answering, medical reasoning, multi-choice questions, and medical conversations. (3) Specifically for multi-choice questions in the medical domain, we propose a novel Verification-of-Choice approach for prompting engineering, which significantly enhances the reasoning ability of LLMs. Remarkably, by combining the above approaches, our AntGLM-Med-10B model can outperform the most of LLMs on PubMedQA, including both general and medical LLMs, even when these LLMs have larger model size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑