arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.02233cs.LGmath.OCstat.ML
收藏

视觉Transformer自监督学习的理论分析

A Theoretical Analysis of Self-Supervised Learning for Vision Transformers

University of Pennsylvania(宾夕法尼亚大学);Carnegie Mellon University(卡内基梅隆大学);The Ohio State University(俄亥俄州立大学)

Yu Huang, Zixin Wen, Yuejie Chi, Yingbin Liang

AI总结:

本文通过建模全局与局部特征的不平衡,分析一层softmax视觉Transformer在MAE和对比学习目标下的训练动态,从理论上解释了MAE能同时学习全局与局部特征而CL偏向全局特征的现象。

摘要:

自监督学习已成为计算机视觉的基石,主要分为基于重建的方法(如掩码自编码器MAE)和判别式方法(如对比学习CL)。近期的实证观察表明,MAE和CL捕获了不同类型的表示:CL倾向于关注全局模式,而MAE能够同时熟练地捕获全局信息和细微的局部信息。尽管最近有大量实证研究试图阐明这一差异,但理论理解仍然有限,尤其是在主导架构视觉Transformer(ViTs)方面。在本文中,为了提供严格的见解,我们通过考虑两种空间特征来建模视觉数据分布:占主导地位的全局特征和相对微小的局部特征,并研究这些特征之间不平衡的影响。我们使用梯度下降分析了一层基于softmax的ViTs在MAE和CL目标上的训练动态。我们的分析表明,随着特征不平衡程度的变化,使用MAE目标训练的ViTs能够有效学习全局和局部特征以实现接近最优的重建,而使用CL训练的ViTs即使在轻度不平衡下也主要偏向全局特征。这些结果为实证研究中观察到的MAE和CL的不同行为提供了理论解释。

英文摘要:

Self-supervised learning has become a cornerstone in computer vision, primarily divided into reconstruction-based methods like masked autoencoders (MAE) and discriminative methods such as contrastive learning (CL). Recent empirical observations reveal that MAE and CL capture different types of representations: CL tends to focus on global patterns, while MAE adeptly captures both global and subtle local information simultaneously. Despite a flurry of recent empirical investigations to shed light on this difference, theoretical understanding remains limited, especially on the dominant architecture vision transformers (ViTs). In this paper, to provide rigorous insights, we model the visual data distribution by considering two types of spatial features: dominant global features and comparatively minuscule local features, and study the impact of imbalance among these features. We analyze the training dynamics of one-layer softmax-based ViTs on both MAE and CL objectives using gradient descent. Our analysis shows that as the degree of feature imbalance varies, ViTs trained with the MAE objective effectively learn both global and local features to achieve near-optimal reconstruction, while the CL-trained ViTs favor predominantly global features, even under mild imbalance. These results provide a theoretical explanation for distinct behaviors of MAE and CL observed in empirical studies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑