FlashMemory-DeepSeek-V4: 通过前瞻稀疏注意力实现闪电索引超长上下文
FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention
AI总结：
提出前瞻稀疏注意力（LSA），基于DeepSeek-V4架构的神经记忆索引器，通过预测未来上下文需求仅保留关键KV块，在超长上下文场景下将物理KV缓存压缩至全上下文的13.5%，同时保持或略微提升下游准确率。
摘要：
传统大语言模型在解码过程中保持完整的KV缓存，导致超长上下文服务出现严重的GPU内存瓶颈。在本报告中，我们提出前瞻稀疏注意力（LSA），一种基于DeepSeek-V4架构构建的神经记忆索引器驱动的新型推理范式。LSA并非被动地关注所有历史令牌，而是主动预测未来的上下文需求，并仅在GPU内存中保留查询关键的KV块。关键的是，我们通过无骨干的解耦训练策略实例化该架构。通过将索引器制定为标准双编码器架构，我们使用标准检索训练框架独立训练它，而无需将庞大的骨干模型加载到GPU内存中。我们证明这种“少即是多”的范式显著最大化服务效率，同时在依赖长期全局记忆的任务中充当有效的注意力去噪器。在主要的长上下文评估套件（例如LongBench-v2、LongMemEval和RULER）中，FM-DS-V4将平均物理KV缓存占用压缩至全上下文基线的仅13.5%，同时一致地保持或略微提升下游准确率（平均绝对边际+0.6%）。关键的是，在极端500K规模下，FlashMemory将物理KV缓存开销抑制超过90%，而不会破坏骨干的核心推理能力。
英文摘要：
Conventional LLMs keep the full KV cache loaded during decoding, causing a severe GPU memory bottleneck for ultra-long context serving. In this report, we propose \textbf{Lookahead Sparse Attention (LSA)}, a novel inference paradigm powered by a Neural Memory Indexer built upon the DeepSeek-V4 architecture. Rather than passively attending to all historical tokens, LSA proactively predicts future context demands and preserves only the query-critical KV chunks in the GPU memory. Crucially, we instantiate this architecture via a \textbf{backbone-free decoupled training} strategy. By formulating the indexer as a standard dual-encoder architecture, we train it independently using standard retrieval training frameworks without ever loading the massive backbone model into GPU memory. We demonstrate that this ``less is more'' paradigm significantly maximizes serving efficiency while acting as an effective attention denoiser in tasks that rely on long-term global memory. Across primary long-context evaluation suites (e.g., LongBench-v2, LongMemEval, and RULER), \texttt{FM-DS-V4} compresses the average physical KV cache footprint down to merely 13.5\% of the full-context baseline, while consistently preserving or slightly elevating downstream accuracy (+0.6\% absolute margin on average). At 1M context, per-decode-token compute drops to 0.30$\times$ of the baseline and GPU KV cache shrinks by 90\% (3.73$\to$0.37 GB), translating into \textbf{2.8$\times$ aggregate throughput and 2.7$\times$ concurrency gains} in PD-disaggregated serving on 8$\times$H20 GPUs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
