EcoFormer：线性复杂度的节能注意力
EcoFormer: Energy-Saving Attention with Linear Complexity
摘要：
Transformer是一个转换框架，它对序列数据进行建模，并在广泛的任务上取得了卓越的性能，但具有较高的计算和能源成本。为了提高其效率，一个流行的选择是通过二进制化压缩模型，将浮点值限制为二进制值，以节省资源消耗，因为廉价的逐位操作显着。然而，现有的二值化方法只着眼于最小化输入分布的统计信息损失，而忽略了两两相似性建模的核心注意力。为此，我们提出了一个新的二进制化范式定制的高维softmax注意力通过内核化哈希，称为EcoFormer，映射到低维的二进制代码在汉明空间中的原始查询和密钥。学习核哈希函数，以自监督的方式匹配从注意力地图中提取的地面真实相似关系。基于二进制码的内积与Hamming距离的等价性以及矩阵乘法的结合性，我们可以将注意力表示为二进制码的点积来近似线性复杂度。此外，查询和键的紧凑二进制表示使我们能够用简单的累加来取代大多数昂贵的乘法累加操作，从而节省边缘设备上的大量片上能量占用。在视觉和语言任务上的大量实验表明，EcoFormer在消耗更少资源的同时，始终实现与标准注意力相当的性能。例如，基于PVTv 2-B 0和ImageNet-1 K，Ecoformer实现了73%的片上能源足迹减少，与标准注意力相比，性能仅下降0.33%。代码可在https://github.com/ziplab/EcoFormer上获得。
英文摘要：
Transformer is a transformative framework that models sequential data and has achieved remarkable performance on a wide range of tasks, but with high computational and energy cost. To improve its efficiency, a popular choice is to compress the models via binarization which constrains the floating-point values into binary ones to save resource consumption owing to cheap bitwise operations significantly. However, existing binarization methods only aim at minimizing the information loss for the input distribution statistically, while ignoring the pairwise similarity modeling at the core of the attention. To this end, we propose a new binarization paradigm customized to high-dimensional softmax attention via kernelized hashing, called EcoFormer, to map the original queries and keys into low-dimensional binary codes in Hamming space. The kernelized hash functions are learned to match the ground-truth similarity relations extracted from the attention map in a self-supervised way. Based on the equivalence between the inner product of binary codes and the Hamming distance as well as the associative property of matrix multiplication, we can approximate the attention in linear complexity by expressing it as a dot-product of binary codes. Moreover, the compact binary representations of queries and keys enable us to replace most of the expensive multiply-accumulate operations in attention with simple accumulations to save considerable on-chip energy footprint on edge devices. Extensive experiments on both vision and language tasks show that EcoFormer consistently achieves comparable performance with standard attentions while consuming much fewer resources. For example, based on PVTv2-B0 and ImageNet-1K, Ecoformer achieves a 73% on-chip energy footprint reduction with only a 0.33% performance drop compared to the standard attention. Code is available at https://github.com/ziplab/EcoFormer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
