FMMformer:通过分解近场和远场注意力的高效和灵活的Transformer
FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention
摘要:
我们提出FMMformers,一类高效和灵活的Transformers的灵感来自著名的快速多极子方法(FMM)加速相互作用的粒子模拟。FMM将粒子间的相互作用分解为近场和远场分量,分别进行直接计算和粗粒度计算。类似地,FMMformer将注意力分解为近场注意力和远场注意力,通过带状矩阵对近场注意力进行建模,通过低秩矩阵对远场注意力进行建模。计算FMMformer的注意矩阵需要相对于序列长度的计算时间和存储器占用的线性复杂度。相比之下,标准Transformers遭受二次复杂性。我们分析并验证了FMMformer在Long Range Arena和语言建模基准测试中相对于标准Transformer的优势。FMMformer甚至可以在精度方面远远超过标准Transformer。例如,FMMformer在五个长距离竞技场任务上实现$60.74\%$的平均分类准确度,这显著优于标准Transformer的$58.70\%$的平均准确度。
英文摘要:
We propose FMMformers, a class of efficient and flexible transformers inspired by the celebrated fast multipole method (FMM) for accelerating interacting particle simulation. FMM decomposes particle-particle interaction into near-field and far-field components and then performs direct and coarse-grained computation, respectively. Similarly, FMMformers decompose the attention into near-field and far-field attention, modeling the near-field attention by a banded matrix and the far-field attention by a low-rank matrix. Computing the attention matrix for FMMformers requires linear complexity in computational time and memory footprint with respect to the sequence length. In contrast, standard transformers suffer from quadratic complexity. We analyze and validate the advantage of FMMformers over the standard transformer on the Long Range Arena and language modeling benchmarks. FMMformers can even outperform the standard transformer in terms of accuracy by a significant margin. For instance, FMMformers achieve an average classification accuracy of $60.74\%$ over the five Long Range Arena tasks, which is significantly better than the standard transformer's average accuracy of $58.70\%$.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。