使用未掩码令牌学习驱动更强的视觉学习器
Learning with Unmasked Tokens Drives Stronger Vision Learners
AI总结：
针对掩码图像建模因仅关注掩码令牌导致编码器注意力跨度受限的问题，提出将未掩码令牌纳入训练以提供更广泛的上下文监督，显著提升了模型在分类与分割等任务上的性能。
摘要：
掩码图像建模（MIM）已成为领先的自监督学习策略。诸如掩码自编码器（MAE）之类的MIM通过随机掩码输入令牌供编码器处理，并由解码器将掩码令牌重建为输入，从而学习强表示。然而，MIM预训练的编码器通常表现出有限的注意力跨度，这归因于MIM仅专注于回归掩码令牌，可能阻碍编码器的更广泛上下文学习。为了解决这一局限性，我们通过将未掩码令牌明确纳入训练过程来改进MIM。具体而言，我们的方法使编码器能够从更广泛的上下文监督中学习，允许未掩码令牌在解码器重建掩码令牌时体验更广泛的上下文。因此，编码后的未掩码令牌配备了丰富的上下文信息，使掩码令牌能够利用增强的未掩码令牌进行MIM。结果表明，这种简单的改进训练出了更具判别性的表示，在ImageNet-1K上使用ViT-B实现了84.2%的top-1准确率，提升了0.6个百分点。我们将这一成功归因于增强的预训练方法，奇异值谱和注意力分析也证实了这一点。最后，我们的模型在下游语义分割和细粒度视觉分类任务以及多种鲁棒性评估指标上均取得了显著的性能提升。代码可在https://github.com/naver-ai/lut获取。
英文摘要：
Masked image modeling (MIM) has become a leading self-supervised learning strategy. MIMs such as Masked Autoencoder (MAE) learn strong representations by randomly masking input tokens for the encoder to process, with the decoder reconstructing the masked tokens to the input. However, MIM pre-trained encoders often exhibit a limited attention span, attributed to MIM's sole focus on regressing masked tokens only, which may impede the encoder's broader context learning. To tackle the limitation, we improve MIM by explicitly incorporating unmasked tokens into the training process. Specifically, our method enables the encoder to learn from broader context supervision, allowing unmasked tokens to experience broader contexts while the decoder reconstructs masked tokens. Thus, the encoded unmasked tokens are equipped with extensive contextual information, empowering masked tokens to leverage the enhanced unmasked tokens for MIM. As a result, our simple remedy trains more discriminative representations revealed by achieving 84.2% top-1 accuracy with ViT-B on ImageNet-1K with 0.6%p gain. We attribute the success to the enhanced pre-training method, as evidenced by the singular value spectrum and attention analyses. Finally, our models achieve significant performance gains at the downstream semantic segmentation and fine-grained visual classification tasks; and on diverse robust evaluation metrics. Code is available at https://github.com/naver-ai/lut
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
