面向医学问答的领域特定语言模型的持续训练与微调
Continuous Training and Fine-tuning for Domain-Specific Language Models in Medical Question Answering
AI总结：
本文通过持续训练和指令微调将 Llama 2 快速适配到中文医学领域，在 10 亿 token 医学语料和 5.4 万条医师考试数据上训练后，模型性能可比肩 GPT-3.5-turbo 且计算成本更低，为其他专业领域模型训练提供了模板。
摘要：
大型语言模型展现出令人期待的通才能力，但往往缺乏面向领域特定任务的专业知识。从基础模型出发开发领域专家，可以在不产生高昂训练成本的情况下支持一系列应用。本工作展示了一种利用持续训练和指令微调将 Llama 2 基础模型快速适配到中文医学领域的方法。我们首先在来自中文医学参考文献的 10 亿个 token 上进行持续训练，以教授相关词汇和知识。随后，模型在来源于中国国家医师资格考试的 54,000 个示例上进行微调。在中文医学数据上的实验证实了该方法的有效性，所得到的模型性能可与 GPT-3.5-turbo 相媲美，同时使用的计算资源远少于后者。由此产生的领域特定模型可用于多种中文医学应用。更广泛地看，这为在预训练模型缺乏所需专业知识的领域（如法律、科学和工程）中进行大型语言模型的领域特定训练提供了一个模板。
英文摘要：
Large language models exhibit promising general capabilities but often lack specialized knowledge for domain-specific tasks. Developing domain experts from a base model enables a range of applications without prohibitive training costs. This work demonstrates a method using continuous training and instruction fine-tuning to rapidly adapt Llama 2 base models to the Chinese medical domain. We first conduct continuous training on 1B tokens from Chinese medical references to teach relevant vocabulary and knowledge. The models are then fine-tuned on 54K examples sourced from the Chinese National Medical Licensing Examination. Experiments on Chinese medical data confirm the effectiveness of this approach, producing a model comparable to GPT-3.5-turbo while using way less computational resource. The resulting domain-specific model could be useful for various Chinese medical applications. More broadly, this provides a template for domain-specific training of large language models in areas where pre-trained models lack the required expertise, such as law, science, and engineering.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
