变形Token可构建强大的掩码图像模型
Morphing Tokens Draw Strong Masked Image Models
AI总结：
针对掩码图像建模中监督信号空间不一致阻碍表征学习的问题，提出动态Token变形方法，动态聚合token生成上下文化目标，在几乎无额外成本下提升MIM效果，多项基准及下游任务表现优于现有最优方法。
摘要：
掩码图像建模（MIM）已成为预训练Vision Transformer（ViT）的一种极具前景的方法。MIM以逐token的方式预测被掩码的token，以恢复从图像中分词得到、或由视觉-语言模型等预训练模型生成的目标信号。尽管使用分词器或预训练模型是可行的，但它们即便对相邻token也往往会提供空间不一致的监督信号，阻碍模型学习具有判别性的表征。我们的初步研究发现了监督信号中的空间不一致性，并指出解决该问题可提升表征学习效果。基于这一洞见，我们提出动态Token变形（DTM）——一种在保留上下文的同时动态聚合token以生成上下文化目标的新方法，从而有望降低空间不一致性。DTM可兼容多种自监督学习（SSL）框架；我们的实验显示其显著提升了MIM效果，且几乎不会带来额外训练成本。该方法通过使用空间一致性更强的目标优化MIM训练，从更低的损失值可看出训练趋势得到改善。在ImageNet-1K和ADE20K上的实验证明了DTM的优越性，其性能超越了复杂的现有最优MIM方法。此外，在iNaturalist等下游任务上的迁移学习评估及大量实证研究，均验证了DTM的有效性。
英文摘要：
Masked image modeling (MIM) has emerged as a promising approach for pre-training Vision Transformers (ViTs). MIMs predict masked tokens token-wise to recover target signals that are tokenized from images or generated by pre-trained models like vision-language models. While using tokenizers or pre-trained models is viable, they often offer spatially inconsistent supervision even for neighboring tokens, hindering models from learning discriminative representations. Our pilot study identifies spatial inconsistency in supervisory signals and suggests that addressing it can improve representation learning. Building upon this insight, we introduce Dynamic Token Morphing (DTM), a novel method that dynamically aggregates tokens while preserving context to generate contextualized targets, thereby likely reducing spatial inconsistency. DTM is compatible with various SSL frameworks; we showcase significantly improved MIM results, barely introducing extra training costs. Our method facilitates MIM training by using more spatially consistent targets, resulting in improved training trends as evidenced by lower losses. Experiments on ImageNet-1K and ADE20K demonstrate DTM's superiority, which surpasses complex state-of-the-art MIM methods. Furthermore, the evaluation of transfer learning on downstream tasks like iNaturalist, along with extensive empirical studies, supports DTM's effectiveness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
