CL-MAE：课程学习掩码自编码器
CL-MAE: Curriculum-Learned Masked Autoencoders
AI总结：
本文提出 CL-MAE，利用可学习掩码模块在 MAE 自监督训练中生成由易到难的重建课程，并在 ImageNet 及五个下游任务上验证其优于 MAE 的表示学习效果。
摘要：
掩码图像建模已被证明是一种强大的预训练任务，可用于生成能够有效泛化到多个下游任务的鲁棒表示。通常，该方法会随机掩码输入图像中的图像块（token），且掩码策略在训练过程中保持不变。本文提出一种课程学习方法，通过更新掩码策略来持续提升自监督重建任务的复杂度。我们推测，随着任务复杂度逐渐增加，模型能够学习到更复杂且更具可迁移性的表示。为此，我们引入了一种新颖的可学习掩码模块，该模块能够生成不同复杂度的掩码，并将所提模块集成到掩码自编码器（MAE）中。该模块与 MAE 联合训练，同时在训练过程中调整自身行为：从 MAE 的伙伴（优化相同的重建损失）转变为对手（优化相反的损失），并经历一个中性状态。这些行为之间的转换是平滑的，由一个与掩码模块重建损失相乘的因子进行调节。由此产生的训练过程形成了由易到难的课程。我们在 ImageNet 上训练课程学习掩码自编码器（CL-MAE），结果表明其相比 MAE 展现出更优的表示学习能力。在五个下游任务上的实验结果证实了我们的推测，表明课程学习能够成功用于掩码自编码器的自监督训练。我们在 https://github.com/ristea/cl-mae 发布代码。
英文摘要：
Masked image modeling has been demonstrated as a powerful pretext task for generating robust representations that can be effectively generalized across multiple downstream tasks. Typically, this approach involves randomly masking patches (tokens) in input images, with the masking strategy remaining unchanged during training. In this paper, we propose a curriculum learning approach that updates the masking strategy to continually increase the complexity of the self-supervised reconstruction task. We conjecture that, by gradually increasing the task complexity, the model can learn more sophisticated and transferable representations. To facilitate this, we introduce a novel learnable masking module that possesses the capability to generate masks of different complexities, and integrate the proposed module into masked autoencoders (MAE). Our module is jointly trained with the MAE, while adjusting its behavior during training, transitioning from a partner to the MAE (optimizing the same reconstruction loss) to an adversary (optimizing the opposite loss), while passing through a neutral state. The transition between these behaviors is smooth, being regulated by a factor that is multiplied with the reconstruction loss of the masking module. The resulting training procedure generates an easy-to-hard curriculum. We train our Curriculum-Learned Masked Autoencoder (CL-MAE) on ImageNet and show that it exhibits superior representation learning capabilities compared to MAE. The empirical results on five downstream tasks confirm our conjecture, demonstrating that curriculum learning can be successfully used to self-supervise masked autoencoders. We release our code at https://github.com/ristea/cl-mae.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
