AttMEMO：在大内存系统上利用记忆化加速Transformer
AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems
AI总结：
AttMEMO利用大内存系统和记忆化技术，通过语义相似性识别加速Transformer自注意力计算，在几乎不损失准确率的情况下平均降低22%推理延迟。
摘要：
Transformer模型因其卓越的推理准确率和推理吞吐量而广受欢迎。然而，Transformer计算密集，导致推理时间较长。现有关于Transformer推理加速的工作存在局限性，这些局限性源于对Transformer架构的修改或对专用硬件的需求。在本文中，我们发现了利用记忆化来加速Transformer中自注意力机制的机会，且无需上述限制。基于一个独特的观察，即不同推理序列之间的注意力计算存在丰富的相似性，我们构建了一个利用新兴大内存系统的记忆化数据库。我们引入了一种新颖的嵌入技术来寻找语义相似的输入，以识别计算相似性。我们还引入了一系列技术，如内存映射和选择性记忆化，以避免内存复制和不必要的开销。我们实现了平均22%的推理延迟降低（最高达68%），且推理准确率损失可忽略不计。
英文摘要：
Transformer models gain popularity because of their superior inference accuracy and inference throughput. However, the transformer is computation-intensive, causing a long inference time. The existing works on transformer inference acceleration have limitations caused by either the modification of transformer architectures or the need of specialized hardware. In this paper, we identify the opportunities of using memoization to accelerate the self-attention mechanism in transformers without the above limitations. Built upon a unique observation that there is rich similarity in attention computation across inference sequences, we build a memoization database that leverages the emerging big memory system. We introduce a novel embedding technique to find semantically similar inputs to identify computation similarity. We also introduce a series of techniques such as memory mapping and selective memoization to avoid memory copy and unnecessary overhead. We enable 22% inference-latency reduction on average (up to 68%) with negligible loss in inference accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
