arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.05262cs.CV
收藏

位置嵌入需要独立的层归一化

Position Embedding Needs an Independent Layer Normalization

Peking University(北京大学);Peng Cheng Laboratory(鹏城实验室)

Runyi Yu, Zhennan Wang, Yinhuai Wang, Kehan Li, Yian Zhao, Jian Zhang, Guoli Song, Jie Chen

摘要:

由于自注意操作的排列不变性,位置嵌入(PE)对于视觉Transformer(VT)是至关重要的。通过使用重新参数化和可视化分析VT中每个编码器层的输入和输出,我们发现默认的PE连接方法(简单地将PE和补丁嵌入添加在一起)对令牌嵌入和PE进行相同的仿射变换,这限制了PE的表现力,从而限制了VT的性能。为了克服这一局限性,我们提出了一种简单,有效,和强大的方法。具体来说,我们为每个层的令牌嵌入和PE提供两个独立的层归一化,并将它们添加在一起作为每个层的多头自注意模块的输入。由于该方法允许模型自适应地调整不同层的PE信息,我们将其称为层自适应位置嵌入,简称为LaPE。大量的实验表明,LaPE可以改善不同类型的PE的各种VT,并使VT对PE类型具有鲁棒性。例如,LaPE将Cifar 10上的ViT-Lite的准确率提高了0.94%,将Cifar 100上的CCT的准确率提高了0.98%,将ImageNet-1 K上的DeiT的准确率提高了1.72%,考虑到LaPE带来的可忽略不计的额外参数、内存和计算成本,这是非常了不起的。该代码可在https://github.com/Ingrid725/LaPE上公开获取。

英文摘要:

The Position Embedding (PE) is critical for Vision Transformers (VTs) due to the permutation-invariance of self-attention operation. By analyzing the input and output of each encoder layer in VTs using reparameterization and visualization, we find that the default PE joining method (simply adding the PE and patch embedding together) operates the same affine transformation to token embedding and PE, which limits the expressiveness of PE and hence constrains the performance of VTs. To overcome this limitation, we propose a simple, effective, and robust method. Specifically, we provide two independent layer normalizations for token embeddings and PE for each layer, and add them together as the input of each layer's Muti-Head Self-Attention module. Since the method allows the model to adaptively adjust the information of PE for different layers, we name it as Layer-adaptive Position Embedding, abbreviated as LaPE. Extensive experiments demonstrate that LaPE can improve various VTs with different types of PE and make VTs robust to PE types. For example, LaPE improves 0.94% accuracy for ViT-Lite on Cifar10, 0.98% for CCT on Cifar100, and 1.72% for DeiT on ImageNet-1K, which is remarkable considering the negligible extra parameters, memory and computational cost brought by LaPE. The code is publicly available at https://github.com/Ingrid725/LaPE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑