arXiv 2304.105202023-09-14cs.CVcs.AIcs.LG

对比微调：一点帮助让掩码自编码器遗忘

Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget

Faculty of Computer Science, University of Vienna, Vienna, Austria（维也纳大学计算机科学学院）；UniVie Doctoral School Computer Science, University of Vienna（维也纳大学计算机科学博士学院）；Institute of Advanced Research in Artificial Intelligence (IARAI)（高级人工智能研究所）

Johannes Lehner, Benedikt Alkin, Andreas Fürst, Elisabeth Rumetshofer, Lukas Miklautz, Sepp Hochreiter

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出 MAE-CT 方法，结合 MIM 的效率与 ID 的对象聚焦能力，通过 NNCLR 目标在预训练 MAE 顶层诱导语义聚类，无需标签且计算开销低，在 ViT 模型上多项指标达 SOTA。

摘要：

掩码图像建模（MIM）方法，如 Masked Autoencoders (MAE)，能高效学习输入的丰富表示。然而，为了适应下游任务，它们需要大量标注数据，因为其丰富特征不仅编码了对象，还编码了相关性较低的图像背景。相比之下，实例判别（ID）方法专注于对象。在本研究中，我们探讨了如何将 MIM 的效率与可扩展性，与 ID 在缺乏大量标注数据时执行下游分类的能力相结合。为此，我们提出了 Masked Autoencoder Contrastive Tuning (MAE-CT)，这是一种顺序方法，利用最近邻对比学习（NNCLR）目标的隐式聚类，在预训练 MAE 的最顶层诱导抽象。MAE-CT 调整丰富特征，使其在不使用任何标签的情况下形成对象的语义聚类。值得注意的是，MAE-CT 不依赖人工设计的增强，且通常仅使用最小增强（裁剪和翻转）即可达到最佳性能。此外，MAE-CT 计算高效，与 MAE 重新训练相比，最多仅需 10% 的额外开销。应用于大型和超大 Vision Transformer (ViT) 模型时，MAE-CT 在线性探测、k-NN 和低样本分类准确率以及无监督聚类准确率上，均超越了此前在 ImageNet 上训练的自监督方法。使用 ViT-H/16，MAE-CT 在线性探测中达到了 82.2% 的最新最优水平。

英文摘要：

Masked Image Modeling (MIM) methods, like Masked Autoencoders (MAE), efficiently learn a rich representation of the input. However, for adapting to downstream tasks, they require a sufficient amount of labeled data since their rich features code not only objects but also less relevant image background. In contrast, Instance Discrimination (ID) methods focus on objects. In this work, we study how to combine the efficiency and scalability of MIM with the ability of ID to perform downstream classification in the absence of large amounts of labeled data. To this end, we introduce Masked Autoencoder Contrastive Tuning (MAE-CT), a sequential approach that utilizes the implicit clustering of the Nearest Neighbor Contrastive Learning (NNCLR) objective to induce abstraction in the topmost layers of a pre-trained MAE. MAE-CT tunes the rich features such that they form semantic clusters of objects without using any labels. Notably, MAE-CT does not rely on hand-crafted augmentations and frequently achieves its best performances while using only minimal augmentations (crop & flip). Further, MAE-CT is compute efficient as it requires at most 10% overhead compared to MAE re-training. Applied to large and huge Vision Transformer (ViT) models, MAE-CT excels over previous self-supervised methods trained on ImageNet in linear probing, k-NN and low-shot classification accuracy as well as in unsupervised clustering accuracy. With ViT-H/16 MAE-CT achieves a new state-of-the-art in linear probing of 82.2%.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。