持续低秩缩放点积注意力
Continual Low-Rank Scaled Dot-product Attention
AI总结：
针对Transformer缩放点积注意力计算内存开销大、难应用于受限流数据处理的问题，提出基于Nyström近似的适用于持续推理的低秩缩放点积注意力，实验中运算量最多降三个数量级且性能持平。
摘要：
Transformer凭借其在序列处理中捕捉数据关联的能力得到广泛应用，在各类静态任务中取得了巨大成功。然而，其核心组件即Scaled Dot-product Attention（缩放点积注意力）的计算与内存开销通常被忽视，这使得它们难以应用于存在响应延迟、计算与内存资源限制的流数据处理场景。已有研究提出了降低Transformer计算成本的方法，包括低秩近似、注意力稀疏化以及面向Continual Inference（持续推理）的高效公式设计。本文提出了一种基于Nyström近似的Scaled Dot-product Attention新公式，适用于持续推理场景。在Online Audio Classification（在线音频分类）和Online Action Detection（在线动作检测）任务的实验中，所提出的Continual Scaled Dot-product Attention（持续缩放点积注意力）与原始Transformer相比，运算量最多可降低三个数量级，同时保持了同类竞争模型的预测性能。
英文摘要：
Transformers are widely used for their ability to capture data relations in sequence processing, with great success for a wide range of static tasks. However, the computational and memory footprint of their main component, i.e., the Scaled Dot-product Attention, is commonly overlooked. This makes their adoption in applications involving stream data processing with constraints in response latency, computational and memory resources infeasible. Some works have proposed methods to lower the computational cost of Transformers, i.e. low-rank approximations, sparsity in attention, and efficient formulations for Continual Inference. In this paper, we introduce a new formulation of the Scaled Dot-product Attention based on the Nyström approximation that is suitable for Continual Inference. In experiments on Online Audio Classification and Online Action Detection tasks, the proposed Continual Scaled Dot-product Attention can lower the number of operations by up to three orders of magnitude compared to the original Transformers while retaining the predictive performance of competing models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
