MKA:用于高效长上下文推理的内存键注意
MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning
AI总结:
本文提出MKA机制,通过多级KV缓存和动态路由提升长上下文推理效率,FastMKA在保持准确率的同时显著提升训练速度和评估效率。
摘要:
随着长上下文语言模型的重要性增加,维护和关注大型键/值(KV)缓存的成本迅速上升,成为训练和推理的主要瓶颈。尽管多查询注意(MQA)和多潜在注意(MLA)等方法通过共享或压缩KV特征来减少内存,但通常会牺牲表示质量或引入运行时开销。我们提出了内存键注意(MKA),一种层次化注意机制,整合了多级KV缓存(本地、会话和长期)并动态学习在它们之间路由注意。我们进一步引入了路由融合的MKA(FastMKA),一种广播路由变体,在注意计算前融合内存源以提高效率。在不同序列长度的实验中,FastMKA实现了良好的准确性-效率权衡:与MLA相比, perplexity相当,但训练吞吐量快5倍,评估延迟低1.8倍。这些结果突显了MKA作为高效长上下文注意的实用且可扩展的框架。
英文摘要:
As long-context language modeling becomes increasingly important, the cost of maintaining and attending to large Key/Value (KV) caches grows rapidly, becoming a major bottleneck in both training and inference. While prior works such as Multi-Query Attention (MQA) and Multi-Latent Attention (MLA) reduce memory by sharing or compressing KV features, they often trade off representation quality or incur runtime overhead. We propose Memory-Keyed Attention (MKA), a hierarchical attention mechanism that integrates multi-level KV caches (local, session, and long-term) and learns to route attention across them dynamically. We further introduce Route-Fused MKA (FastMKA), a broadcast-routed variant that fuses memory sources before attention computation for improved efficiency. Experiments on different sequence lengths show that FastMKA achieves a favorable accuracy-efficiency trade-off: comparable perplexity to MLA while achieving up to 5x faster training throughput and 1.8x lower evaluation latency. These results highlight MKA as a practical and extensible framework for efficient long-context attention.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。