通过训练协议在视觉变换器中诱导空间局部性
Inducing Spatial Locality in Vision Transformers through the Training Protocol
AI总结：
研究通过对比不同训练协议，发现CutMix能提升视觉变换器早期层的注意力局部性，降低MAD值，表明CutMix促进局部注意力的产生。
摘要：
我们研究了是否可以通过训练协议在从头训练的视觉变换器（ViT）的早期层中诱导空间局部性，而无需大规模预训练。在CIFAR-10、CIFAR-100和Tiny-ImageNet上，我们比较了基线协议与现代协议（AutoAugment/ColorJitter、CutMix和Label Smoothing），通过均值注意力距离（MAD）和归一化熵来表征每个注意力头。在所有三个数据集中，现代协议在早期层产生更局部和更集中的注意力；在CIFAR-100上，最小MAD从0.316（基线）降至0.008（现代）。为了确定这种效果的来源，我们在CIFAR-100上进行了消融研究，分别添加或移除每个组件。结果表明CutMix是实验中的决定性组件：所有包含CutMix的条件均显示MAD为0.024，而所有不包含CutMix的条件仍保持在MAD 0.210。AutoAugment和Label Smoothing对局部性无独立影响。总体而言，这些发现表明，由CutMix诱导的从部分图像区域进行分类的压力，可以促进视觉变换器中局部注意力的出现。
英文摘要：
We investigate whether the training protocol can induce spatial locality in the early layers of a Vision Transformer (ViT) trained from scratch, without large-scale pretraining. Keeping the architecture and optimization procedure fixed, we compare a Baseline protocol with a Modern protocol (AutoAugment/ColorJitter, CutMix, and Label Smoothing) on CIFAR-10, CIFAR-100, and Tiny-ImageNet, characterizing each attention head via Mean Attention Distance (MAD) and normalized entropy. Across all three datasets, the Modern protocol produces more local and more concentrated attention in early layers; on CIFAR-100, the minimum MAD drops from 0.316 (Baseline) to 0.008 (Modern). To identify the source of this effect, we conduct an ablation study on CIFAR-100 by adding or removing each component individually. The results identify CutMix as the determining component within our experiments: all conditions with CutMix exhibit MAD 0.024, while all conditions without CutMix remain at MAD 0.210. AutoAugment and Label Smoothing show no independent effect on locality. Taken together, these findings suggest that the pressure to classify from partial image regions, induced by CutMix, can promote the emergence of local attention in Vision Transformers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
