arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.15697q-bio.GNcs.AIcs.CEcs.CL
收藏

dnaGrinder:一个轻量级且高容量的基因组基础模型

dnaGrinder: a lightweight and high-capacity genomic foundation model

Hong Kong Jockey Club STEM Laboratory of Genomics and AI in Healthcare(香港赛马会基因组学与医疗人工智能 STEM 实验室);Department of Health Technology and Informatics(健康技术与信息学系);Department of Data Science and Artificial Intelligence(数据科学与人工智能系);Department of Computing(计算机系);The Hong Kong Polytechnic University(香港理工大学)

Qihang Zhao, Chi Zhang, Weixiong Zhang

AI总结:

针对基因组基础模型在长程依赖、计算成本与性能间的权衡难题,提出轻量级高效模型dnaGrinder,在降低计算成本的同时支持超长序列输入,性能超越Nucleotide Transformer和DNABERT-2等主流模型。

摘要:

理解并解释基因组序列中编码的复杂信息,仍然是生物学研究和临床应用中的一项重大挑战。在此背景下,大型语言模型研究的最新进展推动了仅编码器和仅解码器基础模型的开发,这些模型旨在解码DNA序列中的复杂信息。然而,一些问题依然存在,特别是在高效管理基因组序列固有的长程依赖性、有效表示核苷酸变异,以及与大型模型架构和广泛预训练数据集相关的巨大计算成本方面。当前的基因组基础模型通常面临一个关键的权衡:性能一般的较小模型与性能提升的大型模型。为了应对这些挑战,我们引入了dnaGrinder,一个独特且高效的基因组基础模型。dnaGrinder擅长管理基因组序列内的长程依赖性,同时在最小化计算成本的同时不牺牲性能。它取得的结果不仅可与Nucleotide Transformer和DNABERT-2等领先的DNA模型相媲美,而且往往更胜一筹。此外,dnaGrinder专为在工作站级GPU上轻松微调而设计,可处理超过17,000个token的输入长度。在单个高性能GPU上,它支持长度超过140,000个token的序列,使其成为基础生物学研究和临床应用的高效且易于使用的工具。

英文摘要:

The task of understanding and interpreting the complex information encoded within genomic sequences remains a grand challenge in biological research and clinical applications. In this context, recent advancements in large language model research have led to the development of both encoder-only and decoder-only foundation models designed to decode intricate information in DNA sequences. However, several issues persist, particularly regarding the efficient management of long-range dependencies inherent in genomic sequences, the effective representation of nucleotide variations, and the considerable computational costs associated with large model architectures and extensive pretraining datasets. Current genomic foundation models often face a critical tradeoff: smaller models with mediocre performance versus large models with improved performance. To address these challenges, we introduce dnaGrinder, a unique and efficient genomic foundation model. dnaGrinder excels at managing long-range dependencies within genomic sequences while minimizing computational costs without compromising performance. It achieves results that are not just comparable but often superior to leading DNA models such as Nucleotide Transformer and DNABERT-2. Furthermore, dnaGrinder is designed for easy fine-tuning on workstation-grade GPUs, accommodating input lengths exceeding 17,000 tokens. On a single high-performance GPU, it supports sequences longer than 140,000 tokens, making it a highly efficient and accessible tool for both basic biological research and clinical applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑