arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.00672cs.LGcs.AIcs.CLcs.CV
收藏

通过令牌扩展实现Transformer的通用高效训练

A General and Efficient Training for Transformer via Token Expansion

School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Tencent Youtu Lab(腾讯云图实验室);Xiamen University(厦门大学);Beihang University(北京航空航天大学);Key Laboratory of Advanced Theory and Application in Statistics and Data Science - MOE(先进理论与应用统计科学教育部重点实验室)

Wenxuan Huang, Yunhang Shen, Jiao Xie, Baochang Zhang, Gaoqi He, Ke Li, Xing Sun, Shaohui Lin

AI总结:

本文提出令牌扩展(ToE)方法,通过“初始化-扩展-合并”流程保持特征分布完整性,在不改变超参数和架构的前提下实现ViTs约1.3倍无损训练加速,并可无缝集成到现有训练框架中。

摘要:

视觉Transformer(ViTs)的卓越性能通常需要极大的训练成本。现有方法试图加速ViTs的训练,但通常忽视方法的通用性并导致精度下降。同时,它们破坏了原始Transformer的训练一致性,包括超参数、架构和策略的一致性,这使得它们难以广泛应用于不同的Transformer网络。本文提出了一种新颖的令牌增长方案——令牌扩展(Token Expansion,简称ToE),以实现ViTs的一致训练加速。我们引入了一种“初始化-扩展-合并”流程,以保持原始Transformer中间特征分布的完整性,防止训练过程中关键可学习信息的丢失。ToE不仅可以无缝集成到Transformer(例如DeiT和LV-ViT)的训练和微调过程中,而且对高效训练框架(例如EfficientTrain)同样有效,无需改变原始训练超参数、架构,也无需引入额外的训练策略。大量实验表明,ToE以无损方式实现ViTs训练约1.3倍的加速,甚至相比全令牌训练基线还能获得性能提升。代码可在https://github.com/Osilly/TokenExpansion获取。

英文摘要:

The remarkable performance of Vision Transformers (ViTs) typically requires an extremely large training cost. Existing methods have attempted to accelerate the training of ViTs, yet typically disregard method universality with accuracy dropping. Meanwhile, they break the training consistency of the original transformers, including the consistency of hyper-parameters, architecture, and strategy, which prevents them from being widely applied to different Transformer networks. In this paper, we propose a novel token growth scheme Token Expansion (termed ToE) to achieve consistent training acceleration for ViTs. We introduce an "initialization-expansion-merging" pipeline to maintain the integrity of the intermediate feature distribution of original transformers, preventing the loss of crucial learnable information in the training process. ToE can not only be seamlessly integrated into the training and fine-tuning process of transformers (e.g., DeiT and LV-ViT), but also effective for efficient training frameworks (e.g., EfficientTrain), without twisting the original training hyper-parameters, architecture, and introducing additional training strategies. Extensive experiments demonstrate that ToE achieves about 1.3x faster for the training of ViTs in a lossless manner, or even with performance gains over the full-token training baselines. Code is available at https://github.com/Osilly/TokenExpansion .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑