arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.20154cs.LG
收藏

表示学习中的谱鬼怪:从成分分析到自监督学习

Spectral Ghost in Representation Learning: A Unified View from Component Analysis to Self-Supervised Learning

Google DeepMind(谷歌DeepMind);Georgia Tech(佐治亚理工学院);Harvard University(哈佛大学);University of Alberta(阿尔伯塔大学)

Bo Dai, Na Li, Dale Schuurmans

AI总结:

本文提出了一种基于谱表示的统一框架,用于理解和改进表示学习,通过理论分析揭示现有SSL算法的本质并推动更高效的应用方法。

摘要:

自监督学习(SSL)通过释放未标记数据的潜力,提高了实际应用的实证表现。具体而言,SSL从大量的未标记数据中提取表示,这些表示可以转移到具有有限数据的大量下游任务中。表示学习在各种应用中的显著改进引起了越来越多的关注,导致了各种截然不同的自监督学习目标用于表示提取,伴随着多种学习过程,但缺乏清晰且统一的理解。这种缺乏阻碍了表示学习的持续发展,导致理论理解的缺失,高效算法设计的原则不明确,以及在实践中使用表示学习方法的正当性不成立。统一框架的紧迫性进一步由表示学习方法的快速增长所推动。在本文中,我们因此被迫开发表示学习的原理性基础。我们首先从谱表示的角度理论上研究表示的充分性,这揭示了现有成功的SSL算法的谱本质,并为理解和分析的统一框架铺平了道路。这样的框架工作也激发了更高效且易于使用的表示学习算法的发展,通过在实际应用中采用原理性方法。

英文摘要:

Self-supervised learning (SSL) extracts representations from massive unlabeled data that transfer to a variety of downstream tasks. Its empirical success has produced a dizzying array of objectives and procedures, yet a clear understanding of what representation is desired, and why these seemingly different methods work, is still missing. In this paper, we develop a unified spectral framework for representation learning. We first show that a representation is sufficient for arbitrary downstream prediction with a linear head if it spans the spectral decomposition of the pointwise mutual information, which can be extracted purely from unlabeled data. This view organizes representation learning along two axes, the parametrization of the spectral representation (linear, energy-based, denoising, nonlinear, latent-variable, and multi-modal) and the learning objective for fitting it, under which a broad range of existing methods, including SimCLR, MoCo, BYOL, Barlow Twins, VICReg, DINO, MAE, JEPA, BERT, GPT, and CLIP, as well as classic PCA, CCA, Laplacian eigenmaps, LLE, and SNE, are recovered as particular combinations. The framework also clarifies optimization behaviors: the batch-size requirement is determined by whether the objective is unbiased, a consistent surrogate, or biased, rather than by contrastive vs. non-contrastive losses. Finally, the framework inspires new scalable algorithms and extends spectral representations to causal inference and reinforcement learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑