xTrimoPGLM:用于解读蛋白质语言的统一100B规模预训练Transformer
xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein
AI总结:
针对现有蛋白质语言模型无法同时兼顾理解与生成任务的问题,提出统一预训练框架xTrimoPGLM,以1000亿参数规模训练,在多项基准上性能领先,还支持蛋白质从头与可编程生成,推动蛋白质基础模型发展。
摘要:
蛋白质语言模型在从蛋白质序列中学习生物信息方面已取得显著成效。然而,现有大多数模型受限于自编码或自回归预训练目标,难以同时处理蛋白质理解与生成任务。我们提出一种统一的蛋白质语言模型xTrimoPGLM,通过创新的预训练框架同时应对这两类任务。我们的核心技术贡献是探索了两类目标的兼容性与联合优化潜力,由此形成了一套训练策略,支持以1000亿参数、1万亿训练token的空前规模训练xTrimoPGLM。大量实验表明:1)xTrimoPGLM在四大类共18个蛋白质理解基准上显著优于其他先进基线模型。该模型还能实现蛋白质结构的原子分辨率观测,由此构建的先进3D结构预测模型性能超越现有基于语言模型的工具。2)xTrimoPGLM不仅能遵循天然蛋白质的原理从头生成蛋白质序列,还能在经过筛选的序列上进行监督微调(SFT)后实现可编程生成。这些结果凸显了xTrimoPGLM在蛋白质序列理解与生成方面的强大能力与通用性,为蛋白质科学领域基础模型的发展格局做出了贡献。
英文摘要:
Protein language models have shown remarkable success in learning biological information from protein sequences. However, most existing models are limited by either autoencoding or autoregressive pre-training objectives, which makes them struggle to handle protein understanding and generation tasks concurrently. We propose a unified protein language model, xTrimoPGLM, to address these two types of tasks simultaneously through an innovative pre-training framework. Our key technical contribution is an exploration of the compatibility and the potential for joint optimization of the two types of objectives, which has led to a strategy for training xTrimoPGLM at an unprecedented scale of 100 billion parameters and 1 trillion training tokens. Our extensive experiments reveal that 1) xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories. The model also facilitates an atomic-resolution view of protein structures, leading to an advanced 3D structural prediction model that surpasses existing language model-based tools. 2) xTrimoPGLM not only can generate de novo protein sequences following the principles of natural ones, but also can perform programmable generation after supervised fine-tuning (SFT) on curated sequences. These results highlight the substantial capability and versatility of xTrimoPGLM in understanding and generating protein sequences, contributing to the evolving landscape of foundation models in protein science.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。