通过阅读理解使大型语言模型适应特定领域
Adapting Large Language Models to Domains via Reading Comprehension
AI总结：
本文提出将原始领域语料库转换为阅读理解文本的预训练方法，在保留模型问答能力的同时注入领域知识，7B模型性能媲美BloombergGPT-50B等更大规模模型，并提升通用基准表现。
摘要：
我们探索了在特定领域语料库上继续预训练如何影响大型语言模型，发现基于原始语料库的训练虽然赋予了模型领域知识，但极大地损害了其用于问答的提示能力。受人类通过阅读理解学习的启发——阅读后练习能提高基于所学知识回答问题的能力——我们提出了一种将原始语料库转换为阅读理解文本的简单方法。每篇原始文本都被扩充了一系列与其内容相关的任务。我们的方法具有高度可扩展性，适用于任何预训练语料库，在生物医学、金融和法律三个不同领域的各种任务中持续提升了性能。值得注意的是，我们的7B语言模型实现了与BloombergGPT-50B等规模大得多的特定领域模型相媲美的性能。此外，我们证明了特定领域的阅读理解文本甚至可以提高模型在通用基准上的表现，展现了开发跨越更多领域的通用模型的潜力。我们的模型、代码和数据可在https://github.com/microsoft/LMOps获取。
英文摘要：
We explore how continued pre-training on domain-specific corpora influences large language models, revealing that training on the raw corpora endows the model with domain knowledge, but drastically hurts its prompting ability for question answering. Taken inspiration from human learning via reading comprehension--practice after reading improves the ability to answer questions based on the learned knowledge--we propose a simple method for transforming raw corpora into reading comprehension texts. Each raw text is enriched with a series of tasks related to its content. Our method, highly scalable and applicable to any pre-training corpora, consistently enhances performance across various tasks in three different domains: biomedicine, finance, and law. Notably, our 7B language model achieves competitive performance with domain-specific models of much larger scales, such as BloombergGPT-50B. Furthermore, we demonstrate that domain-specific reading comprehension texts can improve the model's performance even on general benchmarks, showing the potential to develop a general model across even more domains. Our model, code, and data are available at https://github.com/microsoft/LMOps.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
