PolaFormer:面向视觉Transformer的极性感知线性注意力
PolaFormer: Polarity-aware Linear Attention for Vision Transformers
AI总结:
针对线性注意力因非负约束和指数近似导致信息丢失与高熵的问题,提出极性感知线性注意力机制PolaFormer,显式建模同号和异号query-key交互并采用可学习幂函数降低熵,有效提升了视觉任务的表达能力与效率。
摘要:
线性注意力已成为基于softmax注意力的有力替代方案,它利用核化特征图将复杂度从序列长度的二次方降至线性。然而,特征图上的非负约束以及近似中使用的松弛指数函数,相较于原始query-key点积会导致显著的信息丢失,致使注意力图的判别性降低且熵更高。为了解决query-key对中由负值缺失引发的交互问题,我们提出了一种极性感知线性注意力机制,该机制显式建模同号和异号的query-key交互,确保关系信息的全面覆盖。此外,为恢复注意力图的尖锐特性,我们提供了理论分析,证明存在一类元素级函数(具有正的一阶和二阶导数)能够降低注意力分布的熵。为简化并认识到各维度的独特贡献,我们采用可学习的幂函数进行重缩放,使强弱注意力信号得以有效分离。大量实验表明,提出的PolaFormer提升了各种视觉任务的性能,通过高达4.6%的幅度增强了表达能力和效率。
英文摘要:
Linear attention has emerged as a promising alternative to softmax-based attention, leveraging kernelized feature maps to reduce complexity from quadratic to linear in sequence length. However, the non-negative constraint on feature maps and the relaxed exponential function used in approximation lead to significant information loss compared to the original query-key dot products, resulting in less discriminative attention maps with higher entropy. To address the missing interactions driven by negative values in query-key pairs, we propose a polarity-aware linear attention mechanism that explicitly models both same-signed and opposite-signed query-key interactions, ensuring comprehensive coverage of relational information. Furthermore, to restore the spiky properties of attention maps, we provide a theoretical analysis proving the existence of a class of element-wise functions (with positive first and second derivatives) that can reduce entropy in the attention distribution. For simplicity, and recognizing the distinct contributions of each dimension, we employ a learnable power function for rescaling, allowing strong and weak attention signals to be effectively separated. Extensive experiments demonstrate that the proposed PolaFormer improves performance on various vision tasks, enhancing both expressiveness and efficiency by up to 4.6%.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。