基于显著性的自适应掩码：重新审视Token动态以增强预训练
Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training
AI总结：
该研究提出基于显著性的自适应掩码（SBAM）方法及自适应掩码比例（AMR）策略，通过优先考虑token显著性提升掩码图像建模预训练性能，降低对掩码比例的敏感性，在ImageNet-1K上显著优于现有最优方法。
摘要：
本文提出了基于显著性的自适应掩码（SBAM），这是一种新颖且低成本的方法，通过优先考虑token显著性，显著提升了掩码图像建模（MIM）方法的预训练性能。我们的方法对掩码比例的变化具有鲁棒性，有效缓解了现有方法中常见的性能不稳定问题。这降低了基于MIM的预训练对掩码比例的敏感性，进而使我们能够提出一种自适应策略，为每个数据样本设置“定制化”的掩码比例，这是现有方法无法实现的。为实现这一目标，我们提出了自适应掩码比例（AMR）策略，该策略根据token显著性动态调整每张图像独特内容的掩码比例。实验表明，在ImageNet-1K数据集上，我们的方法相较于基于掩码的预训练的当前最优水平有显著提升。
英文摘要：
In this paper, we introduce Saliency-Based Adaptive Masking (SBAM), a novel and cost-effective approach that significantly enhances the pre-training performance of Masked Image Modeling (MIM) approaches by prioritizing token salience. Our method provides robustness against variations in masking ratios, effectively mitigating the performance instability issues common in existing methods. This relaxes the sensitivity of MIM-based pre-training to masking ratios, which in turn allows us to propose an adaptive strategy for `tailored' masking ratios for each data sample, which no existing method can provide. Toward this goal, we propose an Adaptive Masking Ratio (AMR) strategy that dynamically adjusts the proportion of masking for the unique content of each image based on token salience. We show that our method significantly improves over the state-of-the-art in mask-based pre-training on the ImageNet-1K dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
