本文提出 Scala 框架,通过训练时激活多子网、隔离激活和尺度协调,使单个 ViT 无需改结构即可支持多种宽度的灵活推理,并在 ImageNet-1K 上以更少参数超过现有方法。
摘要:
Vision Transformers(ViT)以其可扩展性著称。在本工作中,我们致力于缩减 ViT 的规模,以适应资源约束动态变化的环境。我们观察到,较小的 ViT 本质上是较大 ViT 在不同宽度下的子网络。因此,我们提出了一个名为 Scala 的通用框架,使单个网络能够表示多个具有灵活推理能力的较小 ViT,这与 ViT 沿宽度变化的固有设计相一致。具体而言,Scala 在训练期间激活若干子网,引入 Isolated Activation(隔离激活)以将最小子网络与其他子网解耦,并利用 Scale Coordination(尺度协调)确保每个子网络获得简化、稳定且准确的学习目标。在不同任务上的全面实验验证表明,仅需一次训练,Scala 即可在不修改原始 ViT 结构的情况下学习可缩减宽度的表示,并达到与 Separate Training(单独训练)相匹配的性能。与现有技术相比,Scala 在 ImageNet-1K 上以更少的参数实现了平均 1.6% 的提升。
英文摘要:
Vision Transformers (ViT) is known for its scalability. In this work, we target to scale down a ViT to fit in an environment with dynamic-changing resource constraints. We observe that smaller ViTs are intrinsically the sub-networks of a larger ViT with different widths. Thus, we propose a general framework, named Scala, to enable a single network to represent multiple smaller ViTs with flexible inference capability, which aligns with the inherent design of ViT to vary from widths. Concretely, Scala activates several subnets during training, introduces Isolated Activation to disentangle the smallest sub-network from other subnets, and leverages Scale Coordination to ensure each sub-network receives simplified, steady, and accurate learning objectives. Comprehensive empirical validations on different tasks demonstrate that with only one-shot training, Scala learns slimmable representation without modifying the original ViT structure and matches the performance of Separate Training. Compared with the prior art, Scala achieves an average improvement of 1.6% on ImageNet-1K with fewer parameters.
DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
Monash University(莫纳什大学);Oxford University(牛津大学);Alibaba Group(阿里巴巴集团);Sun Yat-sen University(中山大学);DarkMatter AI Research(暗物质人工智能研究院);Qilu University of Technology(齐鲁工业大学);RMIT University(皇家墨尔本理工大学)
FiT: Flexible Vision Transformer for Diffusion Model
Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Sydney University(悉尼大学);The University of Hong Kong(香港大学)
GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers
Codec Avatars Lab, Meta(Meta编码器动画实验室);University of Tübingen(图宾根大学);Max Planck Institute for Informatics, Saarland Informatics Campus(马克斯·普朗克信息学院,萨尔兰信息校园)
FM-ViT: Flexible Modal Vision Transformers for Face Anti-Spoofing
Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);Baidu Research(百度研究院);Great Bay University(大湾区大学);Mininglamp Academy of Sciences, Mininglamp Technology(明略科技明略科学院);Macau University of Science and Technology(澳门科技大学);Westlake University(西湖大学)
There and Back Again: A Flexible-Frame Transformer for Multi-Exposure Fusion
Nankai International Advanced Research Institute (SHENZHEN·FUTIAN)(南开国际先进研究院(深圳·福田));Peng Cheng Laboratory(鹏城实验室);College of Computer Science, Nankai University(南开大学计算机学院);The Hong Kong Polytechnic University(香港理工大学);OPPO Research Institute(OPPO研究院)
Surface Vision Transformers: Flexible Attention-Based Modelling of Biomedical Surfaces
King’s College London(伦敦国王学院);Washington University Medical School(华盛顿大学医学院);Imperial College London(帝国理工学院);Technical University of Munich(慕尼黑工业大学);University of Edinburgh(爱丁堡大学)