Large language models (LLMs) have demonstrated remarkable open-domain capabilities. LLMs tailored for a domain are typically trained entirely on domain corpus to excel at handling domain-specific tasks. In this work, we explore an alternative strategy of continual pre-training as a means to develop domain-specific LLMs over an existing open-domain LLM. We introduce FinPythia-6.9B, developed through domain-adaptive continual pre-training on the financial domain. Continual pre-trained FinPythia showcases consistent improvements on financial tasks over the original foundational model. We further explore simple but effective data selection strategies for continual pre-training. Our data selection strategies outperform vanilla continual pre-training's performance with just 10% of corpus size and cost, without any degradation on open-domain standard tasks. Our work proposes an alternative solution to building domain-specific LLMs cost-effectively.
TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuning
College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);College of Software, Jilin University(吉林大学软件学院);Fudan University(复旦大学);School of Mathematical and Computer Sciences, Heriot-Watt University(赫瑞-瓦特大学数学与计算机科学学院);Department of Information Engineering and Computer Science, University of Trento(特伦托大学信息工程与计算机科学系)
ELO: Efficient Layer-Specific Optimization for Continual Pretraining of Multilingual LLMs
Seoul National University of Science and Technology(首尔科学技术大学);LG CNS;Korea Advanced Institute of Science and Technology(韩国科学技术院);Hanbat National University(汉 bat 国立大学)
D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models
Alibaba Group(阿里巴巴集团);University of Waterloo(滑铁卢大学);University of Manchester(曼彻斯特大学);QMUL(伦敦玛丽女王大学);The Hong Kong University of Science and Technology(香港科技大学)
Large Language Model Empowered Recommendation Meets All-domain Continual Pre-Training
National University of Singapore(新加坡国立大学);Singapore Management University(新加坡管理大学);Tencent(腾讯);Shandong University(山东大学);City, University London(伦敦城市大学)
Learning Dynamics in Continual Pre-Training for Large Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China(人工智能学院,中国科学院大学,北京);State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院,北京)
Information-Theoretic Complementary Prompts for Improved Continual Text Classification
Mohamed bin Zayed University of Artificial Intelligence(莫莫德·本·扎耶德人工智能大学);Center for Excellence in Brain Science and Intelligence Technology(脑科学与智能技术 excellence 中心);Chinese Academy of Sciences(中国科学院);Tencent AI Lab(腾讯人工智能实验室)