M5：单核苷酸分辨率的全基因组细菌编码器
M5: A Whole Genome Bacterial Encoder at Single Nucleotide Resolution
AI总结：
本文提出细菌全基因组编码器 M5，通过轻量线性注意力机制逼近二次注意力，将上下文长度扩展至数百万单核苷酸，并在单块 A100 GPU 上验证了长序列下的性能提升与注意力近似稳定性。
摘要：
本文描述了一种线性注意力机制，用于将本报告中称为 M5 的仅编码器 Transformer 的上下文长度扩展到数百万个单核苷酸分辨率的基础模型，该模型在细菌全基因组上进行了预训练。所使用的线性注意力机制紧密逼近完整的二次注意力机制，并且在键-查询嵌入维度较低的使用场景下具有简单且轻量级的实现。M5-small 模型完全在一块具有 40GB 显存的 A100 GPU 上进行训练和测试，训练时最长可达 196K 个核苷酸，测试时最长可达 2M 个核苷酸。我们测试了 M5-small 模型的性能，并记录了随着细菌全基因组序列长度的增加而出现的显著性能提升，同时证明了随着序列长度增加所使用的完整多头注意力近似的稳定性。
英文摘要：
A linear attention mechanism is described to extend the context length of an encoder only transformer, called M5 in this report, to a multi-million single nucleotide resolution foundation model pretrained on bacterial whole genomes. The linear attention mechanism used approximates a full quadratic attention mechanism tightly and has a simple and lightweight implementation for the use case when the key-query embedding dimensionality is low. The M5-small model is entirely trained and tested on one A100 GPU with 40gb of memory up to 196K nucleotides during training and 2M nucleotides during testing. We test the performance of the M5-small model and record notable improvements in performance as whole genome bacterial sequence lengths are increased as well as demonstrating the stability of the full multi-head attention approximation used as sequence length is increased.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
