注意力机制的快速蒙特卡罗近似
Fast Monte-Carlo Approximation of the Attention Mechanism
摘要：
我们介绍了蒙特卡罗注意力（MCA），一个随机近似方法，以减少自注意力机制的计算成本在Transformer架构。MCA利用了输入序列中每个标记的重要性随其注意力分数而变化的事实;因此，在编码具有低注意力的标记时，可以容忍一定程度的错误。使用近似矩阵乘法，MCA应用不同的误差范围来编码输入标记，使得具有低注意力分数的标记以宽松的精度计算，而显著元素的误差被最小化。MCA可以与其他注意力优化方案并行操作，并且不需要修改模型。我们研究的理论误差范围，并证明MCA降低注意力的复杂性（在FLOPS）的各种Transformer模型高达11 $\times$在GLUE基准不影响模型的准确性。
英文摘要：
We introduce Monte-Carlo Attention (MCA), a randomized approximation method for reducing the computational cost of self-attention mechanisms in Transformer architectures. MCA exploits the fact that the importance of each token in an input sequence varies with respect to their attention scores; thus, some degree of error can be tolerable when encoding tokens with low attention. Using approximate matrix multiplication, MCA applies different error bounds to encode input tokens such that those with low attention scores are computed with relaxed precision, whereas errors of salient elements are minimized. MCA can operate in parallel with other attention optimization schemes and does not require model modification. We study the theoretical error bounds and demonstrate that MCA reduces attention complexity (in FLOPS) for various Transformer models by up to 11$\times$ in GLUE benchmarks without compromising model accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
