使用具有字节级精度的编码器-解码器基础模型理解DNA的自然语言
Understanding the Natural Language of DNA using Encoder-Decoder Foundation Models with Byte-level Precision
AI总结：
本文提出ENBED基础模型，采用编码器-解码器Transformer架构以字节级精度分析DNA序列，通过掩码语言建模预训练，在增强子识别、错误检测、功能注释及病毒突变生成等任务上显著优于现有方法。
摘要：
本文提出了集成核苷酸字节级编码器-解码器（ENBED）基础模型，利用编码器-解码器Transformer架构以字节级精度分析DNA序列。ENBED使用注意力的次二次实现来开发能够进行序列到序列转换的高效模型，泛化了以往仅有编码器或仅有解码器架构的基因组模型。我们使用掩码语言建模（MLM）利用参考基因组序列预训练基础模型，并将其应用于以下下游任务：（1）增强子、启动子和剪接位点的识别；（2）识别包含碱基调用错配和插入/缺失错误的序列，这优于涉及多个碱基对的分词方案，因为后者失去了以字节级精度分析的能力；（3）基因组序列生物学功能注释的识别；（4）使用编码器-解码器架构生成流感病毒突变，并对照真实世界观察结果进行验证。在每项任务中，我们均展示出相较于现有最优结果的显著提升。
英文摘要：
This paper presents the Ensemble Nucleotide Byte-level Encoder-Decoder (ENBED) foundation model, analyzing DNA sequences at byte-level precision with an encoder-decoder Transformer architecture. ENBED uses a sub-quadratic implementation of attention to develop an efficient model capable of sequence-to-sequence transformations, generalizing previous genomic models with encoder-only or decoder-only architectures. We use Masked Language Modeling to pre-train the foundation model using reference genome sequences and apply it in the following downstream tasks: (1) identification of enhancers, promotors and splice sites, (2) recognition of sequences containing base call mismatches and insertion/deletion errors, an advantage over tokenization schemes involving multiple base pairs, which lose the ability to analyze with byte-level precision, (3) identification of biological function annotations of genomic sequences, and (4) generating mutations of the Influenza virus using the encoder-decoder architecture and validating them against real-world observations. In each of these tasks, we demonstrate significant improvement as compared to the existing state-of-the-art results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
