Protein language models (pLMs) have recently gained significant attention for their ability to uncover relationships between sequence, structure, and function from evolutionary statistics, thereby accelerating therapeutic drug discovery. These models learn from large protein databases that are continuously updated by the biology community and whose dynamic nature motivates the application of continual learning, not only to keep up with the ever-growing data, but also as an opportunity to take advantage of the temporal meta-information that is created during this process. As a result, we introduce the Continual Pretraining of Protein Language Models (CoPeP) benchmark, a novel benchmark for evaluating continual learning approaches on pLMs. Specifically, we curate a sequence of protein datasets derived from the UniProt Knowledgebase spanning a decade and define metrics to assess pLM performance across 31 protein understanding tasks. We evaluate several methods from the continual learning literature, including replay, unlearning, and plasticity-based methods, some of which have never been applied to models and data of this scale. Our findings reveal that incorporating temporal meta-information improves perplexity by up to 7% even when compared to training on data from all tasks jointly. Moreover, even at scale, several continual learning methods outperform naive continual pretraining. The CoPeP benchmark offers an exciting opportunity to study these methods at scale in an impactful real-world application.
ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning
Key Laboratory of High Confidence Software Technologies, Ministry of Education(高可信软件技术重点实验室,教育部);School of Computer Science, Peking University(北京大学计算机学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
A Practitioner's Guide to Continual Multimodal Pretraining
Tübingen AI Center(蒂宾根人工智能中心);University of Tübingen(蒂宾根大学);Helmholtz Munich(亥姆霍兹慕尼黑中心);University of Cambridge(剑桥大学);LAION;Jülich Supercomputing Center (JSC)(于利希超级计算中心);Research Center Jülich (FZJ)(于利希研究中心);Helmholtz Association(亥姆霍兹联合会);Google DeepMind(谷歌DeepMind);Munich Center for ML(慕尼黑机器学习中心);Technical University of Munich(慕尼黑工业大学)
BioBridge: Bridging Proteins and Language for Enhanced Biological Reasoning with LLMs
School of Computer Science and Technology(计算机科学与技术学院);College of Computer Science and Artificial Intelligence(计算机科学与人工智能学院);Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室)
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
Babelscape(巴别景观公司);LAION(LAION组织);TU Darmstadt(达姆施塔特工业大学);UNC Chapel-Hill(北卡罗来纳大学教堂山分校);Detomo Inc.(Detomo公司);CSIRO’s Data61(联邦科学与工业研究组织Data61);Monash University(莫纳什大学);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);Carnegie Mellon University(卡内基梅隆大学);Allen Institute for AI(艾伦人工智能研究所);DeepTensor AB(DeepTensor AB公司);WASP Media & Language(WASP媒体与语言机构);Umeå University(于默奥大学);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);TU Wien(维也纳技术大学);Institute of Science Tokyo(东京科学大学);Hudson Labs(哈德逊实验室);Luleå University of Technology(吕勒奥理工大学);ETH Zurich(苏黎世联邦理工学院);RunPod(RunPod公司);OpenChat(OpenChat机构);RIKEN CCS(理化学研究所计算科学中心);Chattermill AI(Chattermill人工智能公司);ASC27(ASC27机构);University of Massachuset(马萨诸塞大学)