arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2211.09703cs.CVcs.AIcs.LG
收藏

EfficientTrain:探索用于训练视觉骨干网络的广义课程学习

EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones

Tsinghua University(清华大学);Huawei Technologies Ltd.(华为技术有限公司);BAAI(北京智源人工智能研究院)

Yulin Wang, Yang Yue, Rui Lu, Tianjiao Liu, Zhao Zhong, Shiji Song, Gao Huang

AI总结:

本文提出 EfficientTrain,通过从低频、弱增强的易学模式逐步过渡到困难模式的广义课程学习,显著降低多种视觉骨干在 ImageNet 上的训练成本。

摘要:

现代深度网络的优越性能通常伴随着高昂的训练过程。本文提出了一种用于高效训练视觉骨干网络(例如 vision Transformers)的新课程学习方法。我们的工作灵感来自深度网络固有的学习动态:我们通过实验表明,在较早的训练阶段,模型主要学习识别每个样本中某些“更易学”的判别模式,例如图像的较低频率成分以及数据增强之前的原始信息。受这一现象驱动,我们提出一种课程:模型在每个 epoch 始终利用全部训练数据,而该课程开始时仅暴露每个样本的“更易学”模式,随后逐渐引入更困难的模式。为实现这一思想,我们:1)在输入的 Fourier 频谱中引入裁剪操作,使模型能够高效地仅从较低频率成分中学习;2)证明暴露原始图像的特征等价于采用更弱的数据增强;3)将第 1 点和第 2 点结合,并使用贪心搜索算法设计课程学习计划。所得方法 EfficientTrain 简单、通用且异常有效。作为一种即插即用的方法,它在 ImageNet-1K/22K 上将多种流行模型(例如 ResNet、ConvNeXt、DeiT、PVT、Swin 和 CSWin)的实际墙钟时间训练成本降低了 >1.5 倍,且不会牺牲准确率。它对自监督学习(例如 MAE)也有效。代码可在 https://github.com/LeapLabTHU/EfficientTrain 获取。

英文摘要:

The superior performance of modern deep networks usually comes with a costly training procedure. This paper presents a new curriculum learning approach for the efficient training of visual backbones (e.g., vision Transformers). Our work is inspired by the inherent learning dynamics of deep networks: we experimentally show that at an earlier training stage, the model mainly learns to recognize some 'easier-to-learn' discriminative patterns within each example, e.g., the lower-frequency components of images and the original information before data augmentation. Driven by this phenomenon, we propose a curriculum where the model always leverages all the training data at each epoch, while the curriculum starts with only exposing the 'easier-to-learn' patterns of each example, and introduces gradually more difficult patterns. To implement this idea, we 1) introduce a cropping operation in the Fourier spectrum of the inputs, which enables the model to learn from only the lower-frequency components efficiently, 2) demonstrate that exposing the features of original images amounts to adopting weaker data augmentation, and 3) integrate 1) and 2) and design a curriculum learning schedule with a greedy-search algorithm. The resulting approach, EfficientTrain, is simple, general, yet surprisingly effective. As an off-the-shelf method, it reduces the wall-time training cost of a wide variety of popular models (e.g., ResNet, ConvNeXt, DeiT, PVT, Swin, and CSWin) by >1.5x on ImageNet-1K/22K without sacrificing accuracy. It is also effective for self-supervised learning (e.g., MAE). Code is available at https://github.com/LeapLabTHU/EfficientTrain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑