The Position Embedding (PE) is critical for Vision Transformers (VTs) due to the permutation-invariance of self-attention operation. By analyzing the input and output of each encoder layer in VTs using reparameterization and visualization, we find that the default PE joining method (simply adding the PE and patch embedding together) operates the same affine transformation to token embedding and PE, which limits the expressiveness of PE and hence constrains the performance of VTs. To overcome this limitation, we propose a simple, effective, and robust method. Specifically, we provide two independent layer normalizations for token embeddings and PE for each layer, and add them together as the input of each layer's Muti-Head Self-Attention module. Since the method allows the model to adaptively adjust the information of PE for different layers, we name it as Layer-adaptive Position Embedding, abbreviated as LaPE. Extensive experiments demonstrate that LaPE can improve various VTs with different types of PE and make VTs robust to PE types. For example, LaPE improves 0.94% accuracy for ViT-Lite on Cifar10, 0.98% for CCT on Cifar100, and 1.72% for DeiT on ImageNet-1K, which is remarkable considering the negligible extra parameters, memory and computational cost brought by LaPE. The code is publicly available at https://github.com/Ingrid725/LaPE.
Adaptive Cross Batch Normalization for Metric Learning
Amazon Australia(亚马逊澳大利亚);Australian National University(澳大利亚国立大学);University of Adelaide(阿德莱德大学)
度量学习是计算机视觉中的一个基本问题,其中模型被训练为通过排名损失来学习语义上有用的嵌入空间。传统上,排名损失的有效性取决于小批量大小,因此,固有地受到底层硬件的内存约束的限制。虽然简单地累积小批量的嵌入已被证明是有用的(Wang et al. [2020]),但我们表明,确保累积的嵌入是最新的也同样重要。特别是,当可学习参数被更新时,有必要避免当前训练迭代中累积嵌入和特征嵌入之间的代表性漂移。在本文中,我们代表性的漂移模型的分布错位和解决它使用矩匹配。结果是一种简单的方法,用于更新存储的嵌入,以在每次训练迭代时匹配当前嵌入的一阶和二阶矩。在三个流行的图像检索数据集,即SOP,In-Shop和DeepFashion 2上的实验表明,我们的方法在所有场景下都显着提高了性能。
Layer-Aware Position Embeddings for Visual Token Pruning in Multimodal Large Language Models
Tongji University(同济大学);University of California, Santa Cruz(加州大学圣克鲁兹分校);East China Normal University(华东师范大学);Shanghai Eye Disease Prevention and Treatment Center(上海市眼病防治中心)
Local Context Normalization: Revisiting Local Normalization
The University of Texas at El Paso(德克萨斯大学埃尔帕索分校);Microsoft AI for Good Research Lab(微软益世AI研究实验室);Georgia Institute of Technology(佐治亚理工学院);Microsoft Research(微软研究院)