近无损模型压缩实现DNA大语言模型的更长上下文推理
Near-Lossless Model Compression Enables Longer Context Inference in DNA Large Language Models
AI总结：
针对DNA大语言模型长序列推理受自注意力二次复杂度与KV缓存内存限制的问题，提出可嵌入预训练模型的渐进压缩模块FOCUS，实现近无损百倍压缩，将推理复杂度降至近线性，支持消费级GPU上百倍长窗口推理。
摘要：
在海量跨物种DNA语料上训练的DNA大语言模型（LLM）学习到了基因组序列的基本“语法”和演化模式，这使其成为DNA序列建模（尤其是长序列建模）的强大先验。但实际应用中存在两大核心限制：自注意力的二次计算成本，以及自回归解码过程中键值（KV）缓存不断增长的内存需求。这些限制迫使研究者采用固定窗口截断或滑动窗口等启发式方法，这类方法会丢弃远距离信息，损害超长序列上的建模保真度。本文提出FOCUS（面向超长自注意力的特征导向压缩），这是一种可嵌入预训练DNA LLM的渐进式上下文压缩模块。FOCUS将基因组学中成熟的k-mer表示与可学习的分层压缩相结合：以k-mer为粒度插入摘要token，跨多个Transformer层渐进式压缩注意力键和值激活，仅保留跨窗口的摘要KV状态，丢弃普通token的KV。共享边界加窗方案可生成稳定的跨窗口接口，以极小损失传播长程信息。本文在基于Evo-2的DNA LLM上验证FOCUS的效果，该模型在GRCh38 1号染色体上通过自监督训练和随机压缩调度微调，以提升不同压缩比下的鲁棒性。在留出的人类染色体测试集上，FOCUS实现了近无损保真度：将1kb上下文压缩为仅10个摘要token（约100倍压缩）时，平均每个核苷酸的概率偏移仅约0.0004。与无压缩基线相比，FOCUS降低了KV缓存内存占用，将有效推理复杂度从O(N²)转化为近线性O(N)，可在消费级GPU上实现约100倍长的推理窗口，同时保持近无损保真度。
英文摘要：
Trained on massive cross-species DNA corpora, DNA large language models (LLMs) learn the fundamental "grammar" and evolutionary patterns of genomic sequences. This makes them powerful priors for DNA sequence modeling, particularly over long ranges. However, two major constraints hinder their use in practice: the quadratic computational cost of self-attention and the growing memory required for key-value (KV) caches during autoregressive decoding. These constraints force the use of heuristics such as fixed-window truncation or sliding windows, which compromise fidelity on ultra-long sequences by discarding distant information. We introduce FOCUS (Feature-Oriented Compression for Ultra-long Self-attention), a progressive context-compression module that can be plugged into pretrained DNA LLMs. FOCUS combines the established k-mer representation in genomics with learnable hierarchical compression: it inserts summary tokens at k-mer granularity and progressively compresses attention key and value activations across multiple Transformer layers, retaining only the summary KV states across windows while discarding ordinary-token KV. A shared-boundary windowing scheme yields a stationary cross-window interface that propagates long-range information with minimal loss. We validate FOCUS on an Evo-2-based DNA LLM fine-tuned on GRCh38 chromosome 1 with self-supervised training and randomized compression schedules to promote robustness across compression ratios. On held-out human chromosomes, FOCUS achieves near-lossless fidelity: compressing a 1 kb context into only 10 summary tokens (about 100x) shifts the average per-nucleotide probability by only about 0.0004. Compared to a baseline without compression, FOCUS reduces KV-cache memory and converts effective inference scaling from O(N^2) to near-linear O(N), enabling about 100x longer inference windows on commodity GPUs with near-lossless fidelity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
