arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.08545cs.CL
收藏

构建领域特定大语言模型的高效持续预训练

Efficient Continual Pre-training for Building Domain Specific Large Language Models

Amazon Services LLC(亚马逊公司)

Yong Xie, Karan Aggarwal, Aitzaz Ahmad

AI总结:

本研究提出通过持续预训练在现有开放领域LLM上构建领域特定LLM,采用数据选择策略提升效果,降低训练成本。

摘要:

大型语言模型(LLMs)在开放领域展现了显著的能力。针对特定领域的LLMs通常完全在领域语料上进行训练,以在处理领域特定任务上表现出色。在本工作中,我们探索了一种替代策略,即通过在现有开放领域LLM上进行持续预训练来开发领域特定的LLM。我们介绍了通过在金融领域进行领域自适应持续预训练开发出的FinPythia-6.9B。持续预训练的FinPythia在金融任务上相对于原始基础模型展示了持续的改进。我们进一步探讨了简单但有效的数据选择策略用于持续预训练。我们的数据选择策略在仅使用10%语料库大小和成本的情况下,优于传统持续预训练的性能,且在开放领域标准任务上没有退化。我们的工作提出了一种以低成本有效的方式构建领域特定LLM的替代方案。

英文摘要:

Large language models (LLMs) have demonstrated remarkable open-domain capabilities. LLMs tailored for a domain are typically trained entirely on domain corpus to excel at handling domain-specific tasks. In this work, we explore an alternative strategy of continual pre-training as a means to develop domain-specific LLMs over an existing open-domain LLM. We introduce FinPythia-6.9B, developed through domain-adaptive continual pre-training on the financial domain. Continual pre-trained FinPythia showcases consistent improvements on financial tasks over the original foundational model. We further explore simple but effective data selection strategies for continual pre-training. Our data selection strategies outperform vanilla continual pre-training's performance with just 10% of corpus size and cost, without any degradation on open-domain standard tasks. Our work proposes an alternative solution to building domain-specific LLMs cost-effectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑