arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.04786cs.CVcs.LG
收藏

用于灵活推理的切片 Vision Transformer

Slicing Vision Transformer for Flexible Inference

Snap Inc.(Snap公司);Northeastern University(东北大学);Meta

Yitian Zhang, Huseyin Coskun, Xu Ma, Huan Wang, Ke Ma, Xi, Chen, Derek Hao Hu, Yun Fu

AI总结:

本文提出 Scala 框架,通过训练时激活多子网、隔离激活和尺度协调,使单个 ViT 无需改结构即可支持多种宽度的灵活推理,并在 ImageNet-1K 上以更少参数超过现有方法。

摘要:

Vision Transformers(ViT)以其可扩展性著称。在本工作中,我们致力于缩减 ViT 的规模,以适应资源约束动态变化的环境。我们观察到,较小的 ViT 本质上是较大 ViT 在不同宽度下的子网络。因此,我们提出了一个名为 Scala 的通用框架,使单个网络能够表示多个具有灵活推理能力的较小 ViT,这与 ViT 沿宽度变化的固有设计相一致。具体而言,Scala 在训练期间激活若干子网,引入 Isolated Activation(隔离激活)以将最小子网络与其他子网解耦,并利用 Scale Coordination(尺度协调)确保每个子网络获得简化、稳定且准确的学习目标。在不同任务上的全面实验验证表明,仅需一次训练,Scala 即可在不修改原始 ViT 结构的情况下学习可缩减宽度的表示,并达到与 Separate Training(单独训练)相匹配的性能。与现有技术相比,Scala 在 ImageNet-1K 上以更少的参数实现了平均 1.6% 的提升。

英文摘要:

Vision Transformers (ViT) is known for its scalability. In this work, we target to scale down a ViT to fit in an environment with dynamic-changing resource constraints. We observe that smaller ViTs are intrinsically the sub-networks of a larger ViT with different widths. Thus, we propose a general framework, named Scala, to enable a single network to represent multiple smaller ViTs with flexible inference capability, which aligns with the inherent design of ViT to vary from widths. Concretely, Scala activates several subnets during training, introduces Isolated Activation to disentangle the smallest sub-network from other subnets, and leverages Scale Coordination to ensure each sub-network receives simplified, steady, and accurate learning objectives. Comprehensive empirical validations on different tasks demonstrate that with only one-shot training, Scala learns slimmable representation without modifying the original ViT structure and matches the performance of Separate Training. Compared with the prior art, Scala achieves an average improvement of 1.6% on ImageNet-1K with fewer parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑