Cerebras支持的表观基因组语言模型
Epigenomic language models powered by Cerebras
摘要：
Transformer语言模型的大规模自监督预训练推进了自然语言处理领域，并显示出交叉应用于蛋白质和DNA的生物“语言”的前景。使用大型基因组序列语料库学习DNA序列的有效表示可以通过迁移学习加速基因调控和功能模型的开发。然而，为了准确地模拟细胞类型特异性基因调控和功能，不仅需要考虑DNA核苷酸序列中包含的信息，这在细胞类型之间基本不变，而且还需要考虑染色体的局部化学和结构“表观遗传状态”在细胞类型之间如何变化。在这里，我们介绍了一个双向编码器表示从Transformers（BERT）模型，学习表示基于DNA序列和配对的表观遗传状态输入，我们称之为表观基因组BERT（或EBERT）。我们使用掩蔽的语言模型目标在整个人类基因组和127种细胞类型中预训练EBERT。通过与Cerebras Systems的合作，首次实现了用以前庞大的数据集训练这个复杂模型，Cerebras Systems的CS-1系统为所有预训练实验提供了动力。我们展示了EBERT的迁移学习潜力，通过展示细胞类型特异性转录因子结合预测任务的强大性能。我们的微调模型在ENCODE-DREAM基准测试的13个评估数据集中的4个上超过了最先进的性能，并在挑战排行榜上获得了第3名的整体排名。我们探讨了表观遗传数据和任务特定特征增强如何影响迁移学习性能。
英文摘要：
Large scale self-supervised pre-training of Transformer language models has advanced the field of Natural Language Processing and shown promise in cross-application to the biological `languages' of proteins and DNA. Learning effective representations of DNA sequences using large genomic sequence corpuses may accelerate the development of models of gene regulation and function through transfer learning. However, to accurately model cell type-specific gene regulation and function, it is necessary to consider not only the information contained in DNA nucleotide sequences, which is mostly invariant between cell types, but also how the local chemical and structural `epigenetic state' of chromosomes varies between cell types. Here, we introduce a Bidirectional Encoder Representations from Transformers (BERT) model that learns representations based on both DNA sequence and paired epigenetic state inputs, which we call Epigenomic BERT (or EBERT). We pre-train EBERT with a masked language model objective across the entire human genome and across 127 cell types. Training this complex model with a previously prohibitively large dataset was made possible for the first time by a partnership with Cerebras Systems, whose CS-1 system powered all pre-training experiments. We show EBERT's transfer learning potential by demonstrating strong performance on a cell type-specific transcription factor binding prediction task. Our fine-tuned model exceeds state of the art performance on 4 of 13 evaluation datasets from ENCODE-DREAM benchmarks and earns an overall rank of 3rd on the challenge leaderboard. We explore how the inclusion of epigenetic data and task specific feature augmentation impact transfer learning performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
