Large language models (LLMs) are routinely pre-trained on billions of tokens, only to restart the process over again once new data becomes available. A much cheaper and more efficient solution would be to enable the continual pre-training of these models, i.e. updating pre-trained models with new data instead of re-training them from scratch. However, the distribution shift induced by novel data typically results in degraded performance on past data. Taking a step towards efficient continual pre-training, in this work, we examine the effect of different warm-up strategies. Our hypothesis is that the learning rate must be re-increased to improve compute efficiency when training on a new dataset. We study the warmup phase of models pre-trained on the Pile (upstream data, 300B tokens) as we continue to pre-train on SlimPajama (downstream data, 297B tokens), following a linear warmup and cosine decay schedule. We conduct all experiments on the Pythia 410M language model architecture and evaluate performance through validation perplexity. We experiment with different pre-training checkpoints, various maximum learning rates, and various warmup lengths. Our results show that while rewarming models first increases the loss on upstream and downstream data, in the longer run it improves the downstream performance, outperforming models trained from scratch$\unicode{x2013}$even for a large downstream dataset.
TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuning
College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);College of Software, Jilin University(吉林大学软件学院);Fudan University(复旦大学);School of Mathematical and Computer Sciences, Heriot-Watt University(赫瑞-瓦特大学数学与计算机科学学院);Department of Information Engineering and Computer Science, University of Trento(特伦托大学信息工程与计算机科学系)
Large Language Model Empowered Recommendation Meets All-domain Continual Pre-Training
National University of Singapore(新加坡国立大学);Singapore Management University(新加坡管理大学);Tencent(腾讯);Shandong University(山东大学);City, University London(伦敦城市大学)
Continual Pre-training of MoEs: How robust is your router?
Université de Montréal(蒙特利尔大学);Mila – Quebec AI Institute(魁北克人工智能研究所);Concordia University(康科迪亚大学);University of Chicago(芝加哥大学);Capital One(Capital One公司)
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Tsinghua University(清华大学)
Susceptibility of Continual Learning Against Adversarial Attacks
COMSATS University Islamabad(伊斯兰堡COMSATS大学);Bacha Khan University(巴赫汗大学);Chung-Ang University(中央大学);Institute of Space Technology(空间技术研究所);Georgia State University(佐治亚州立大学)
Rehearsal-Free Modular and Compositional Continual Learning for Language Models
Bosch Center for Artificial Intelligence(博世人工智能中心);LMU Munich(慕尼黑大学);Hochschule der Medien(斯图加特媒体大学);Karlsruhe University of Applied Sciences(卡尔斯鲁厄应用技术大学)