arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.07063cs.LG
收藏

InAttention:面向Transformer的线性上下文扩展方法

InAttention: Linear Context Scaling for Transformers

Joseph Eisner

AI总结:

针对Transformer自注意力机制显存随上下文长度二次增长的问题,提出InAttention机制替换自注意力,实现推理时显存线性增长,可在消费级GPU处理长序列,微调能低成本扩展上下文长度,为长距离依赖建模提供可扩展方案。

摘要:

由于自注意力机制的存在,Transformer模型的显存(VRAM)需求随上下文长度呈二次增长。本文对仅解码器Transformer进行改进,用InAttention替换自注意力:该机制让每个token仅关注初始状态,推理阶段显存占用随上下文长度线性增长。基准测试表明,InAttention可显著降低推理时的VRAM用量,支持在消费级GPU上处理长序列。研究证实,微调可高效扩展上下文长度,无需高昂训练成本即可提升长序列任务性能。InAttention为Transformer模型的长距离依赖问题提供了可扩展方案,为后续优化奠定基础。

英文摘要:

VRAM requirements for transformer models scale quadratically with context length due to the self-attention mechanism. In this paper we modify the decoder-only transformer, replacing self-attention with InAttention, which scales linearly with context length during inference by having tokens attend only to initial states. Benchmarking shows that InAttention significantly reduces VRAM usage during inference, enabling handling of long sequences on consumer GPUs. We corroborate that fine-tuning extends context length efficiently, improving performance on long sequences without high training costs. InAttention offers a scalable solution for long-range dependencies in transformer models, paving the way for further optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑