arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.00253cs.LG
收藏

CoPeP:蛋白质语言模型持续预训练基准测试

CoPeP: Benchmarking Continual Pretraining for Protein Language Models

Mila - Quebec AI Institute(魁北克AI研究所);Polytechnique Montréal(蒙特利尔理工学院);Chandar Research Lab(昌达尔研究实验室);CIFAR AI Chair(CIFAR人工智能主席)

Darshan Patil, Pranshu Malviya, Mathieu Reymond, Quentin Fournier, Sarath Chandar

AI总结:

CoPeP基准测试通过评估持续学习方法在蛋白质语言模型中的表现,揭示了整合时间元信息对模型性能的提升效果。

摘要:

蛋白质语言模型(pLMs)最近因其能够从进化统计学中揭示序列、结构和功能之间的关系,从而加速药物发现而受到广泛关注。这些模型从不断更新的大型蛋白质数据库中学习,其动态特性促使应用持续学习,不仅为了跟上日益增长的数据,还利用这一过程中的时间元信息。因此,我们引入了蛋白质语言模型的持续预训练(CoPeP)基准测试,这是一个新的基准,用于评估持续学习方法在pLMs上的性能。具体来说,我们整理了一个从UniProt知识库中获取的十年序列数据集,并定义了评估pLM在31种蛋白质理解任务上的性能的指标。我们评估了几种持续学习方法,包括重放、遗忘和基于可塑性的方法,其中一些方法从未应用于如此大规模的模型和数据。我们的发现表明,即使与联合训练所有任务的数据相比,整合时间元信息也能将困惑度提高高达7%。此外,即使在大规模情况下,几种持续学习方法也优于朴素的持续预训练。CoPeP基准测试为在具有影响力的现实应用中研究这些方法提供了绝佳机会。

英文摘要:

Protein language models (pLMs) have recently gained significant attention for their ability to uncover relationships between sequence, structure, and function from evolutionary statistics, thereby accelerating therapeutic drug discovery. These models learn from large protein databases that are continuously updated by the biology community and whose dynamic nature motivates the application of continual learning, not only to keep up with the ever-growing data, but also as an opportunity to take advantage of the temporal meta-information that is created during this process. As a result, we introduce the Continual Pretraining of Protein Language Models (CoPeP) benchmark, a novel benchmark for evaluating continual learning approaches on pLMs. Specifically, we curate a sequence of protein datasets derived from the UniProt Knowledgebase spanning a decade and define metrics to assess pLM performance across 31 protein understanding tasks. We evaluate several methods from the continual learning literature, including replay, unlearning, and plasticity-based methods, some of which have never been applied to models and data of this scale. Our findings reveal that incorporating temporal meta-information improves perplexity by up to 7% even when compared to training on data from all tasks jointly. Moreover, even at scale, several continual learning methods outperform naive continual pretraining. The CoPeP benchmark offers an exciting opportunity to study these methods at scale in an impactful real-world application.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑