arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2007.06225cs.LGcs.CLcs.DCstat.ML
收藏

ProtTrans:通过自我监督深度学习和高性能计算破解生命语言的代码

ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing

Technical University of Munich(慕尼黑工业大学);Med AI Technology (Wu Xi) Ltd.(无锡明艾智能科技有限公司);Google AI;NVIDIA(英伟达);Oak Ridge National Laboratory(橡树岭国家实验室)
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsin… 展开作者

Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, Burkhard Rost

摘要:

计算生物学和生物信息学从蛋白质序列中提供了巨大的数据金矿,非常适合从NLP中提取的语言模型。这些线性模型以较低的推理成本达到了新的预测前沿。在这里,我们训练了两个自回归模型(Transformer-XL,XLNet)和四个自动编码器模型(BERT,Albert,Electra,T5),这些模型来自UniRef和BFD,包含多达3930亿个氨基酸。LM在Summit超级计算机上进行了训练,使用了5616个GPU和TPU Pod,最多可达1024个核心。非线性约简揭示了来自未标记数据的原始蛋白质LM嵌入捕获了蛋白质序列的一些生物物理特征。我们验证了使用嵌入作为几个后续任务的独占输入的优势。第一个是蛋白质二级结构的每个残基预测(3-状态准确度Q3=81%-87%);第二个是蛋白质亚细胞定位的每个蛋白质预测(10-状态准确度:Q10=81%)和膜与水溶性(2-状态准确度Q2=91%)。对于每残基预测,最具信息性的嵌入(ProtT 5)的转移首次优于最先进的技术,而不使用进化信息,从而绕过昂贵的数据库搜索。总的来说,这些结果意味着蛋白质LMs学习了生命语言的一些语法。为了方便将来的工作,我们在https://github.com/agemagician/ProtTrans上发布了我们的模型。

英文摘要:

Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models taken from NLP. These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The LMs were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw protein LM-embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks. The first was a per-residue prediction of protein secondary structure (3-state accuracy Q3=81%-87%); the second were per-protein predictions of protein sub-cellular localization (ten-state accuracy: Q10=81%) and membrane vs. water-soluble (2-state accuracy Q2=91%). For the per-residue predictions the transfer of the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without using evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that protein LMs learned some of the grammar of the language of life. To facilitate future work, we released our models at https://github.com/agemagician/ProtTrans.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑