arXiv 2206.089542023-06-13cs.CVcs.LG

自监督学习成功背后的图像块嵌入袋（Bag of Image Patch Embedding）

Bag of Image Patch Embedding Behind the Success of Self-Supervised Learning

Meta AI - FAIR（Meta人工智能基础研究实验室）；Center for Data Science, New York University（纽约大学数据科学中心）；Inria（法国国家信息与自动化研究所）；École normale supérieure（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）；Redwood Center（红杉中心）；UC Berkeley（加州大学伯克利分校）；Courant Institute（柯朗研究所）

Yubei Chen, Adrien Bardes, Zengyi Li, Yann LeCun

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究揭示联合嵌入类自监督学习本质是学习反映共现关系的图像块表示，提出BagSSL方法，验证了块表示聚合的有效性与局部性保留特性，为理解自监督表示学习提供新视角。

摘要：

自监督学习（Self-supervised learning, SSL）近年来在图像表示学习方面取得了巨大的实证进展，但我们对这类表示学习背后原理的理解仍然有限。本研究表明，联合嵌入类SSL方法主要学习的是图像块的表示，该反映了图像块的共现关系。我们可以形式化地建立这种与共现建模的关联，这对当前主流的不变性视角形成了补充。实证结果显示，学习固定尺度图像块的表示，并将局部图像块表示聚合为图像表示，能够取得与基线方法相当甚至更优的效果，我们将这一过程命名为BagSSL。即便使用32×32的图像块表示，BagSSL在ImageNet上的top-1线性探测准确率也能达到62%。另一方面，通过多尺度预训练模型我们发现，整图嵌入近似等于局部图像块嵌入的平均值。虽然SSL表示在全局尺度上具有相对不变性，但当我们放大到局部图像块级别的表示时，局部性得以保留。此外，研究表明图像块表示聚合能够大幅提升多种SOTA基线方法的性能。图像块表示更易于理解，本工作为揭开自监督表示学习的神秘面纱迈出了一步。

英文摘要：

Self-supervised learning (SSL) has recently achieved tremendous empirical advancements in learning image representation. However, our understanding of the principle behind learning such a representation is still limited. This work shows that joint-embedding SSL approaches primarily learn a representation of image patches, which reflects their co-occurrence. Such a connection to co-occurrence modeling can be established formally, and it supplements the prevailing invariance perspective. We empirically show that learning a representation for fixed-scale patches and aggregating local patch representations as the image representation achieves similar or even better results than the baseline methods. We denote this process as BagSSL. Even with 32x32 patch representation, BagSSL achieves 62% top-1 linear probing accuracy on ImageNet. On the other hand, with a multi-scale pretrained model, we show that the whole image embedding is approximately the average of local patch embeddings. While the SSL representation is relatively invariant at the global scale, we show that locality is preserved when we zoom into local patch-level representation. Further, we show that patch representation aggregation can improve various SOTA baseline methods by a large margin. The patch representation is considerably easier to understand, and this work makes a step to demystify self-supervised representation learning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。