SpAtten：具有级联令牌和头部剪枝的高效稀疏注意力架构
SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
AI总结：
SpAtten通过级联令牌和头部剪枝及渐进量化，协同设计算法与硬件，在无精度损失下减少DRAM访问10倍，显著提升速度和能效。
摘要：
注意力机制在自然语言处理（NLP）应用中日益流行，展现出优于卷积和循环架构的性能。然而，由于其对输入长度的二次计算复杂度、复杂的数据移动和低算术强度，注意力成为计算瓶颈。此外，现有的神经网络加速器主要专注于优化卷积或循环模型，无法高效支持注意力。在本文中，我们提出了SpAtten，一种高效的算法-架构协同设计，利用令牌稀疏性、头部稀疏性和量化机会来减少注意力计算和内存访问。受人类语言高度冗余性的启发，我们提出了新颖的级联令牌剪枝，以剪除句子中不重要的令牌。我们还提出了级联头部剪枝，以移除不必要的头部。级联剪枝与权重剪枝根本不同，因为注意力机制中没有可训练的权重，且被剪除的令牌和头部是即时选择的。为了在硬件上高效支持它们，我们设计了一种新颖的top-k引擎，以高吞吐量对令牌和头部的重要性分数进行排序。此外，我们提出了渐进量化，首先仅获取最高有效位（MSBs）并执行计算；如果置信度低，则获取最低有效位（LSBs）并重新计算注意力输出，以计算换取内存减少。在30个基准上的广泛实验表明，平均而言，SpAtten在无精度损失的情况下将DRAM访问减少了10.0倍，并相对于A3加速器、MNNFast加速器、TITAN Xp GPU和Xeon CPU分别实现了1.6倍、3.0倍、162倍、347倍的加速，以及1.4倍、3.2倍、1193倍、4059倍的能耗节省。
英文摘要：
The attention mechanism is becoming increasingly popular in Natural Language Processing (NLP) applications, showing superior performance than convolutional and recurrent architectures. However, attention becomes the compution bottleneck because of its quadratic computational complexity to input length, complicated data movement and low arithmetic intensity. Moreover, existing NN accelerators mainly focus on optimizing convolutional or recurrent models, and cannot efficiently support attention. In this paper, we present SpAtten, an efficient algorithm-architecture co-design that leverages token sparsity, head sparsity, and quantization opportunities to reduce the attention computation and memory access. Inspired by the high redundancy of human languages, we propose the novel cascade token pruning to prune away unimportant tokens in the sentence. We also propose cascade head pruning to remove unessential heads. Cascade pruning is fundamentally different from weight pruning since there is no trainable weight in the attention mechanism, and the pruned tokens and heads are selected on the fly. To efficiently support them on hardware, we design a novel top-k engine to rank token and head importance scores with high throughput. Furthermore, we propose progressive quantization that first fetches MSBs only and performs the computation; if the confidence is low, it fetches LSBs and recomputes the attention outputs, trading computation for memory reduction. Extensive experiments on 30 benchmarks show that, on average, SpAtten reduces DRAM access by 10.0x with no accuracy loss, and achieves 1.6x, 3.0x, 162x, 347x speedup, and 1,4x, 3.2x, 1193x, 4059x energy savings over A3 accelerator, MNNFast accelerator, TITAN Xp GPU, Xeon CPU, respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
