抑制性自注意力：在视觉Transformer中锐化注意力

中文摘要 AI 辅助

视觉Transformer（ViTs）在计算机视觉任务中表现出色。但其自注意力机制常将注意力分散到背景区域，依赖虚假相关性而非与对象相关的线索。受生物视觉系统中抑制机制启发，我们提出抑制性自注意力（ISA），一种整合抑制信号以增强特征选择性并抑制虚假响应的新型自注意力。与传统自注意力因softmax归一化仅依赖正注意力值不同，我们的方法保留并利用负注意力分数抑制无关特征，锐化对感兴趣对象的注意力。在包括ImageNet-1k和COCO等多个数据集及多个鲁棒性基准上的实验表明，ISA增强了以对象为中心的选择性，减少了对捷径的依赖，并改善了分布外泛化能力。我们对相关性映射的分析证实，具有ISA的ViTs对与对象相关区域表现出更清晰、更局部化的注意力，同时减少了来自无关（背景）特征的干扰，从而实现更可靠的模型。我们在这个https URL上发布了代码。

英文摘要

Vision Transformers (ViTs) have demonstrated remarkable performance in computer vision tasks. However, their self-attention mechanism often diffuses focus across background regions, relying on spurious correlations rather than object-relevant cues. Inspired by inhibitory mechanisms observed in biological vision systems, we propose the Inhibited Self-Attention (ISA), a novel self-attention that integrates inhibitory signals to enhance feature selectivity and suppress spurious responses. In contrast to conventional self-attention, which relies solely on positive attention values due to softmax normalization, our approach retains and utilizes negative attention scores to suppress irrelevant features and sharpen focus on objects of interest. Experiments across multiple datasets, including ImageNet-1k and COCO, and several robustness benchmarks demonstrate that ISA enhances object-centric selectivity, reduces shortcut reliance, and improves out-of-distribution generalization. Our analysis of relevance maps confirms that ViTs with ISA exhibit sharper, more localized focus on object-relevant regions while reducing distractions from non-relevant (background) features, enabling more reliable models. We release our code at https://github.com/prdvanderwal/inhibited-self-attention

发表机构

• University of Groningen（格罗宁根大学）
• University of Twente（特温特大学）
• Stellenbosch University（斯泰伦博斯大学）
• University of Malta（马耳他大学）

机构由 AI 辅助整理，请以论文原文为准。