训练计算最优的蛋白质语言模型
Training Compute-Optimal Protein Language Models
AI总结:
本研究通过大规模实验揭示了蛋白质语言模型的计算最优训练策略,提出CLM与MLM的缩放定律,并验证了迁移缩放现象,为高效训练提供了指导。
摘要:
我们探索了蛋白质语言模型的最优训练方法,这是生物学研究中一个备受关注但最佳实践指导有限的领域。大多数模型使用大量计算资源进行训练,直到性能提升趋于平稳,主要侧重于增加模型规模,而非优化平衡性能与计算预算的高效计算前沿。我们的研究基于一个包含9.39亿条蛋白质序列的大规模数据集。我们训练了超过300个模型,参数规模从350万到107亿不等,使用的独特令牌数量从50亿到2000亿不等,以研究模型规模、训练令牌数量和目标之间的关系。首先,我们观察到在重复使用常用的Uniref数据库时,因果语言模型(CLM)存在收益递减效应,而掩码语言模型(MLM)则存在过拟合现象。为了解决这一问题,我们在训练集中加入了宏基因组蛋白质序列,以增加多样性并避免性能平台期或过拟合效应。其次,我们针对蛋白质序列数据的特定特征,获得了基于Transformer架构的CLM和MLM的缩放定律。第三,我们观察到从CLM到MLM的迁移缩放现象,进一步通过基于估计的有效迁移令牌的缩放行为证明了迁移的有效性。最后,为了验证我们的缩放定律,我们在下游任务上比较了ESM-2和PROGEN2的大规模版本,评估涵盖了蛋白质生成以及结构和功能相关任务,所有这些都在小于或等于预训练计算预算的条件下进行。
英文摘要:
We explore optimally training protein language models, an area of significant interest in biological research where guidance on best practices is limited. Most models are trained with extensive compute resources until performance gains plateau, focusing primarily on increasing model sizes rather than optimizing the efficient compute frontier that balances performance and compute budgets. Our investigation is grounded in a massive dataset consisting of 939 million protein sequences. We trained over 300 models ranging from 3.5 million to 10.7 billion parameters on 5 to 200 billion unique tokens, to investigate the relations between model sizes, training token numbers, and objectives. First, we observed the effect of diminishing returns for the Causal Language Model (CLM) and that of overfitting for the Masked Language Model~(MLM) when repeating the commonly used Uniref database. To address this, we included metagenomic protein sequences in the training set to increase the diversity and avoid the plateau or overfitting effects. Second, we obtained the scaling laws of CLM and MLM on Transformer, tailored to the specific characteristics of protein sequence data. Third, we observe a transfer scaling phenomenon from CLM to MLM, further demonstrating the effectiveness of transfer through scaling behaviors based on estimated Effectively Transferred Tokens. Finally, to validate our scaling laws, we compare the large-scale versions of ESM-2 and PROGEN2 on downstream tasks, encompassing evaluations of protein generation as well as structure- and function-related tasks, all within less or equivalent pre-training compute budgets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。