arXiv 2405.192662024-11-11cs.CL

PediatricsGPT：作为儿科应用中文医疗助手的大型语言模型

PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications

Academy for Engineering and Technology, Fudan University（复旦大学工程与技术学院）；Tencent Youtu Lab（腾讯优图实验室）；Cognition and Intelligent Technology Laboratory（认知与智能技术实验室）；Engineering Research Center of AI and Robotics, Ministry of Education（教育部人工智能与机器人工程研究中心）；AI and Unmanned Systems Engineering Research Center of Jilin Province（吉林省人工智能与无人系统工程研究中心）

Dingkang Yang, Jinjie Wei, Dongling Xiao, Shunli Wang, Tong Wu, Gang Li, Mingcheng Li, Shuaibing Wang, Jiawei Chen, Yue Jiang, Qingyao Xu, Ke Li, Peng Zhai, Lihua Zhang

备注：: Accepted by NeurIPS 2024. A Technical Report on a Chinese Medical Large Language Model

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文构建了超30万指令的PedCorpus数据集，并提出首个中文儿科大模型助手PediatricsGPT，通过混合指令预训练、全参数SFT、偏好优化及通用-特定专家混合等系统训练流程，在多项评估中持续优于以往中文医疗LLMs。

摘要：

开发智能儿科问诊系统为提高诊断效率提供了广阔前景，特别是在医疗资源匮乏的中国。尽管针对中文医疗的大型语言模型（LLMs）近期取得了进展，但由于指令数据不足和训练过程脆弱，其在儿科应用中的表现仍欠佳。为解决上述问题，本文构建了PedCorpus，这是一个包含超过300,000条多任务指令的高质量数据集，来源于儿科教科书、指南和知识图谱资源，以满足多样化的诊断需求。在精心设计的PedCorpus基础上，我们提出了PediatricsGPT，这是首个基于系统且稳健的训练流程构建的中文儿科LLM助手。在连续预训练阶段，我们引入了混合指令预训练机制，以缓解LLMs在医疗领域适应中内部注入知识的不一致性。紧接着，利用全参数监督微调（SFT）将通用医疗知识模式融入模型中。此后，我们设计了直接跟随偏好优化，以增强生成类似儿科医生的人文回复。在参数高效的二次SFT阶段，提出了一种通用-特定专家混合策略，以解决医疗通才与儿科专业知识掌握之间的能力冲突。基于各项指标、GPT-4以及医生在不同医生下游任务上的评估结果显示，PediatricsGPT持续优于以往的中文医疗LLMs。我们的模型和数据集将为社区发展开源。

英文摘要：

Developing intelligent pediatric consultation systems offers promising prospects for improving diagnostic efficiency, especially in China, where healthcare resources are scarce. Despite recent advances in Large Language Models (LLMs) for Chinese medicine, their performance is sub-optimal in pediatric applications due to inadequate instruction data and vulnerable training procedures. To address the above issues, this paper builds PedCorpus, a high-quality dataset of over 300,000 multi-task instructions from pediatric textbooks, guidelines, and knowledge graph resources to fulfil diverse diagnostic demands. Upon well-designed PedCorpus, we propose PediatricsGPT, the first Chinese pediatric LLM assistant built on a systematic and robust training pipeline. In the continuous pre-training phase, we introduce a hybrid instruction pre-training mechanism to mitigate the internal-injected knowledge inconsistency of LLMs for medical domain adaptation. Immediately, the full-parameter Supervised Fine-Tuning (SFT) is utilized to incorporate the general medical knowledge schema into the models. After that, we devise a direct following preference optimization to enhance the generation of pediatrician-like humanistic responses. In the parameter-efficient secondary SFT phase, a mixture of universal-specific experts strategy is presented to resolve the competency conflict between medical generalist and pediatric expertise mastery. Extensive results based on the metrics, GPT-4, and doctor evaluations on distinct doctor downstream tasks show that PediatricsGPT consistently outperforms previous Chinese medical LLMs. Our model and dataset will be open-source for community development.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。