视觉Transformer中的注意力图幻觉与FFN压缩
Vision Transformer with Attention Map Hallucination and FFN Compaction
AI总结：
提出注意力图幻觉（hMHSA）与FFN压缩（cFFN），减少ViT冗余，实现10%-20%的FLOPs和参数降低，性能保持竞争力。
摘要：
视觉Transformer（ViT）目前在许多视觉任务中占据主导地位。其逐令牌多头自注意力（MHSA）的二次复杂度缺点，已通过令牌稀疏化或（空间或通道上的）维度缩减等方法得到广泛解决。然而，MHSA内部的冗余性以及前馈网络（FFN）的冗余性通常被忽视。为此，我们提出注意力图幻觉和FFN压缩来填补这一空白。具体而言，我们观察到原始ViT中存在相似的注意力图，并提议从其余注意力图中以更廉价的操作幻觉出一半的注意力图，这被称为幻觉多头自注意力（hMHSA）。对于FFN，我们对其隐藏到输出的投影矩阵进行因式分解，并利用重参数化技术增强其能力，使其成为紧凑型FFN（cFFN）。借助我们提出的模块，对于各种基于ViT的骨干网络，包括直通型（DeiT）、混合型（NextViT）和层次型（PVT）结构，浮点运算次数（FLOPs）和参数（Params）减少了10%至20%，同时性能相当有竞争力。
英文摘要：
Vision Transformer(ViT) is now dominating many vision tasks. The drawback of quadratic complexity of its token-wise multi-head self-attention (MHSA), is extensively addressed via either token sparsification or dimension reduction (in spatial or channel). However, the therein redundancy of MHSA is usually overlooked and so is the feed-forward network (FFN). To this end, we propose attention map hallucination and FFN compaction to fill in the blank. Specifically, we observe similar attention maps exist in vanilla ViT and propose to hallucinate half of the attention maps from the rest with much cheaper operations, which is called hallucinated-MHSA (hMHSA). As for FFN, we factorize its hidden-to-output projection matrix and leverage the re-parameterization technique to strengthen its capability, making it compact-FFN (cFFN). With our proposed modules, a 10$\%$-20$\%$ reduction of floating point operations (FLOPs) and parameters (Params) is achieved for various ViT-based backbones, including straight (DeiT), hybrid (NextViT) and hierarchical (PVT) structures, meanwhile, the performances are quite competitive.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
