arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2507.04416cs.CL
收藏

RAT:通过基于分块的序列建模衔接RNN效率与注意力精度

RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling

CLAIRE, EPFL(EPFL的CLAIRE);Google DeepMind(谷歌DeepMind)

Xiuying Wei, Anunay Yadav, Razvan Pascanu, Caglar Gulcehre

AI总结:

针对Transformer注意力计算瓶颈与循环模型长上下文记忆退化问题,提出基于分块的RAT架构,兼顾效率与精度,还提出混合架构进一步提升性能与推理效率。

摘要:

Transformer已成为现代大规模语言模型的基石,但它们对softmax注意力的依赖在训练和推理阶段都构成了计算瓶颈。循环模型具备高效率,但将完整序列压缩为固定大小的整体表示,在长上下文中会出现记忆退化问题,且限制了细粒度检索。为解决该问题,我们提出RAT,这是一种衔接RNN效率与注意力容量的中间设计。RAT将输入划分为多个块,在每个块内应用循环机制处理局部依赖,在块间应用基于softmax的注意力实现长距离交互。该设计缓解了记忆退化问题,支持直接访问远距离token,同时保持了计算效率。实验结果表明,在块大小为16的设置下,对于100K长度的序列,RAT块的训练速度提升7倍,在4K位置的生成速度提升9倍,同时性能与标准注意力相当。我们通过从头训练1.3B参数模型并开展大规模评估(包括短上下文和长上下文基准测试,以及有监督微调(SFT))验证了这一点。我们进一步提出了一种将RAT与局部注意力交错的混合架构。通过将高效的长距离建模与强大的局部交互相结合,该混合设计不仅提升了推理速度、减少了缓存内存占用,还持续提升了性能,展现出整体最优效果。代码可在https://github.com/CLAIRE-Labo/RAT获取。

英文摘要:

Transformers have become the cornerstone of modern large-scale language models, but their reliance on softmax attention poses a computational bottleneck at both training and inference. Recurrent models offer high efficiency, but compressing the full sequence into a fixed-size and holistic representation can suffer from memory degradation in long contexts and limit fine-grained retrieval. To address this, we propose RAT, an intermediate design that bridges the efficiency of RNNs and capacity of attention. RAT partitions the input into chunks, applies recurrence within each chunk for local dependencies, and softmax-based attention across chunks for long-range interactions. This design mitigates memory degradation and enables direct access to distant tokens, while retaining computational efficiency. Empirically, with a chunk size of 16, the RAT block achieves a 7$\times$ improvement in training speed for 100K sequence length and 9$times$ in generation at the 4K position, while maintaining similar performance compared to standard attention. We demonstrate this by training 1.3B parameter models from scratch and performing large-scale evaluations, including short- and long-context benchmarks, as well as supervised fine-tuning~(SFT). We further propose a hybrid architecture that interleaves RAT with local attention. By combining efficient long-range modeling with strong local interactions, this hybrid design not only improves inference speed and reduces cache memory usage, but also consistently enhances performance and shows the overall best results. Code is available at https://github.com/CLAIRE-Labo/RAT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑