TPC-ViT:用于高效视觉Transformer的Token传播控制器
TPC-ViT: Token Propagation Controller for Efficient Vision Transformer
AI总结:
针对ViTs中token冗余假设的局限性,提出token传播控制器(TPC),通过暂停和重启概率控制token的缩减与重用,并引入平滑机制和模型稳定器提升性能,在ImageNet-1K上显著提高了推理速度和分类准确率。
摘要:
视觉Transformer(ViTs)在各种计算机视觉任务上取得了有前景的结果,然而其在输入token数量上的二次复杂度限制了它们的应用,特别是在资源受限的设置中。以前采用渐进式token缩减来应对这一挑战的方法假设,一层的token冗余意味着所有后续层都存在冗余。我们凭经验证明,这种假设通常是不正确的,即在一层中冗余的token在后续层中可能是有用的。我们利用这一关键见解提出了一种新颖的token传播控制器(TPC),它包含两种不同的token分布,即暂停概率和重启概率,分别控制token的缩减和重用,从而实现更高效的token利用。为了改善token分布的估计,我们提出了一种平滑机制,它作为正则化器并帮助消除噪声异常值。此外,为了提高我们提出的TPC的训练稳定性,我们引入了一种模型稳定器,它能够隐式编码局部图像结构并最小化模型训练期间的准确率波动。我们在ImageNet-1K数据集上使用DeiT、LV-ViT和Swin模型展示了广泛的实验结果,以证明我们提出方法的有效性。例如,与基线模型相比,我们提出的方法将DeiT-S的推理速度提高了250%,同时将分类准确率提高了1.0%。
英文摘要:
Vision transformers (ViTs) have achieved promising results on a variety of Computer Vision tasks, however their quadratic complexity in the number of input tokens has limited their application specially in resource-constrained settings. Previous approaches that employ gradual token reduction to address this challenge assume that token redundancy in one layer implies redundancy in all the following layers. We empirically demonstrate that this assumption is often not correct, i.e., tokens that are redundant in one layer can be useful in later layers. We employ this key insight to propose a novel token propagation controller (TPC) that incorporates two different token-distributions, i.e., pause probability and restart probability to control the reduction and reuse of tokens respectively, which results in more efficient token utilization. To improve the estimates of token distributions, we propose a smoothing mechanism that acts as a regularizer and helps remove noisy outliers. Furthermore, to improve the training-stability of our proposed TPC, we introduce a model stabilizer that is able to implicitly encode local image structures and minimize accuracy fluctuations during model training. We present extensive experimental results on the ImageNet-1K dataset using DeiT, LV-ViT and Swin models to demonstrate the effectiveness of our proposed method. For example, compared to baseline models, our proposed method improves the inference speed of the DeiT-S by 250% while increasing the classification accuracy by 1.0%.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。