The superior performance of modern deep networks usually comes with a costly training procedure. This paper presents a new curriculum learning approach for the efficient training of visual backbones (e.g., vision Transformers). Our work is inspired by the inherent learning dynamics of deep networks: we experimentally show that at an earlier training stage, the model mainly learns to recognize some 'easier-to-learn' discriminative patterns within each example, e.g., the lower-frequency components of images and the original information before data augmentation. Driven by this phenomenon, we propose a curriculum where the model always leverages all the training data at each epoch, while the curriculum starts with only exposing the 'easier-to-learn' patterns of each example, and introduces gradually more difficult patterns. To implement this idea, we 1) introduce a cropping operation in the Fourier spectrum of the inputs, which enables the model to learn from only the lower-frequency components efficiently, 2) demonstrate that exposing the features of original images amounts to adopting weaker data augmentation, and 3) integrate 1) and 2) and design a curriculum learning schedule with a greedy-search algorithm. The resulting approach, EfficientTrain, is simple, general, yet surprisingly effective. As an off-the-shelf method, it reduces the wall-time training cost of a wide variety of popular models (e.g., ResNet, ConvNeXt, DeiT, PVT, Swin, and CSWin) by >1.5x on ImageNet-1K/22K without sacrificing accuracy. It is also effective for self-supervised learning (e.g., MAE). Code is available at https://github.com/LeapLabTHU/EfficientTrain.
Progressive Growing of Patch Size: Curriculum Learning for Accelerated and Improved Medical Image Segmentation
School of Computation, Information and Technology, Technical University Munich(技术大学慕尼黑计算、信息与技术学院);Department of Radiation Oncology, TUM School of Medicine, TUM University Hospital rechts der Isar, Technical University of Munich(肿瘤放射治疗科,TUM医学院,TUM大学医院rechts der Isar,技术大学慕尼黑);Institute of Machine Learning in Biomedical Imaging, Helmholtz Munich(生物医学影像中的机器学习研究所,海德堡慕尼黑);Institute of Radiation Medicine, Helmholtz Center Munich(放射医学研究所,海德堡慕尼黑);Munich Center of Machine Learning (MCML)(慕尼黑机器学习中心);School of Biomedical Engineering and Imaging Sciences, King’s College London(生物医学工程与成像科学学院,伦敦国王学院);Departament de Matemàtiques i Informàtica, Barcelona Artificial Intelligence in Medicine Lab (BCN-AIM), Universitat de Barcelona(数学与计算机科学系,巴塞罗那人工智能在医学实验室(BCN-AIM),巴塞罗那大学)
MambaNUT: Nighttime UAV Tracking via Mamba-based Adaptive Curriculum Learning
College of Computer Science and Engineering, Guilin University of Technology(桂林理工大学计算机科学与工程学院);School of Computer Science, Fudan University(复旦大学计算机科学学院);School of Artificial Intelligence, Sun Yat-sen University(中山大学人工智能学院)