arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.26900cs.LG
收藏

SPHERE-JEPA: 均匀嵌入的球面预测

SPHERE-JEPA: Spherical Prediction with Homogeneous Embeddings

Université Paris-Saclay, CNRS, Advanced Track and Trace(巴黎萨克雷大学,国家科学研究中心,先进跟踪与追溯);ENS Paris-Saclay, Centre Borelli, Advanced Track and Trace(巴黎萨克雷高等师范学院,博雷利中心,先进跟踪与追溯);Université Paris-Saclay, CNRS(巴黎萨克雷大学,国家科学研究中心);ENS Paris-Saclay, Centre Borelli(巴黎萨克雷高等师范学院,博雷利中心)

L{é}o Nicollier, Max Dunitz, Marc Pic, Pablo Mus{é}, Enric Meinhardt-Llopis, Gabriele Facciolo

AI总结:

本文提出SPHERE-JEPA框架,通过将Cramér-Wold投影机制调整为强制超球面均匀性而非高斯先验,解决了自监督学习中高斯嵌入导致各向异性k-NN邻域的问题,在纹理检索和ImageNet-1K线性探测上取得显著提升。

摘要:

自监督学习中的一个基本开放问题是明确表征学习表示的最优几何。最近,LeJEPA将各向同性高斯嵌入确定为在欧几里得空间中最小化下游预测风险的最优解。然而,对于支撑在低维流形(如超球面)上的分布,相应问题仍未探索。在这项工作中,我们证明将这种极小极大分析扩展到黎曼流形上的光滑分布会根本性地改变最优解。我们表明,在最坏情况公式下,k近邻和核岭回归都诱导超球面均匀性。更精确地说,我们证明流形上的均匀分布对于k近邻是最优的,而球面上的均匀分布对于使用指数点积核和线性核的核岭回归是最优的。这一理论见解揭示了高斯嵌入的一个根本局限:其非均匀密度导致各向异性的k-NN邻域,严重偏置估计器。为纠正这一点,我们引入了SPHERE-JEPA,一个理论基础的SSL框架。我们调整LeJEPA的Cramér-Wold投影机制以强制超球面均匀性而非高斯先验。实验上,SPHERE-JEPA取得了显著改进,将纹理检索mAP提升了超过6%,同时在标准基准上持续匹配或超越LeJEPA——包括在ImageNet-1K(ViT-B/14)上+1.8%的线性探测增益。

英文摘要:

A fundamental open question in self-supervised learning (SSL) is the explicit characterization of the optimal geometry of the learned representations. Recently, LeJEPA identified isotropic Gaussian embeddings as optimal for minimizing downstream prediction risk in Euclidean spaces. However, the corresponding problem for distributions supported on lower-dimensional manifolds, such as the hypersphere, remains unexplored. In this work, we demonstrate that extending this minimax analysis to smooth distributions on Riemannian manifolds fundamentally changes the optimal solution. We show that, under a worst-case formulation, both k-nearest neighbors and kernel ridge regression induce hyperspherical uniformity. More precisely, we show that uniform distributions on manifolds are optimal for k-nearest neighbors, and that the uniform distribution on the sphere is optimal for kernel ridge regression with both the exponential dot-product kernel and the linear kernel. This theoretical insight reveals a fundamental limitation of Gaussian embeddings: their non-uniform density induces anisotropic k-NN neighborhoods, severely biasing the estimator. To correct this, we introduce SPHERE-JEPA, a theoretically grounded SSL framework. We adapt LeJEPA's Cram{é}r-Wold projection mechanism to enforce hyperspherical uniformity rather than a Gaussian prior. Empirically, SPHERE-JEPA yields significant improvements, boosting texture retrieval mAP by over 6%, while consistently matching or outperforming LeJEPA on standard benchmarks-including a +1.8% linear probing gain on ImageNet-1K (ViT-B/14).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑