学习记住什么:用于内存高效语言模型的自适应概率记忆保留
Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Models
AI总结:
针对Transformer长上下文内存开销大的问题,提出自适应保留机制,通过伯努利门控学习选择性保留token,在30-50%保留率下保持95%以上性能,并显著降低内存、提升吞吐量。
摘要:
Transformer注意力机制的计算复杂度随序列长度呈二次方增长,即O(n^2),这限制了其在长上下文场景中的应用。我们提出自适应保留(Adaptive Retention),一种基于概率的、逐层的token选择机制,该机制在严格的全局预算M下学习应保留哪些表示。保留操作通过伯努利门控建模,并使用Hard-Concrete/变分松弛进行训练,在推理时通过简单的top-M规则强制执行,从而使该方法可微分,并可直接嵌入标准编码器。在分类、抽取式问答和长文档摘要任务中,仅保留30-50%的token即可保持全模型性能的95%以上,同时将峰值内存降低约35-45%,吞吐量提升最高约1.8倍。这种与架构无关的方法在不修改基础注意力机制或任务头的情况下,实现了实用的长上下文效率。
英文摘要:
Transformer attention scales quadratically with sequence length O(n^2), limiting long-context use. We propose Adaptive Retention, a probabilistic, layer-wise token selection mechanism that learns which representations to keep under a strict global budget M. Retention is modeled with Bernoulli gates trained via a Hard-Concrete/variational relaxation and enforced with a simple top-M rule at inference, making the method differentiable and drop-in for standard encoders. Across classification, extractive QA, and long-document summarization, keeping only 30-50% of tokens preserves >= 95% of full-model performance while cutting peak memory by ~35-45% and improving throughput by up to ~1.8x. This architecture-agnostic approach delivers practical long-context efficiency without modifying base attention or task heads.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。