arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01456cs.LGcs.CV
收藏

Rectified LpJEPA:具有稀疏和最大熵表示的联合嵌入预测架构

Rectified LpJEPA: Joint-Embedding Predictive Architectures with Sparse and Maximum-Entropy Representations

New York University(纽约大学);Duke University(杜克大学);University of Toronto(多伦多大学);Brown University(布朗大学)

Yilun Kuang, Yash Dagade, Tim G. J. Rudner, Randall Balestriero, Yann LeCun

AI总结:

提出Rectified Distribution Matching Regularization (RDMReg)损失,通过将表示对齐到Rectified Generalized Gaussian分布,实现稀疏且最大熵的表示,从而改进联合嵌入预测架构(JEPA)的性能。

摘要:

联合嵌入预测架构(JEPA)学习视角不变表示,并采用基于投影的分布匹配来防止崩溃。现有方法将表示正则化为各向同性高斯分布,但固有地偏向密集表示,未能捕捉高效表示中观察到的稀疏性关键特性。我们引入了Rectified Distribution Matching Regularization (RDMReg),这是一种切片双样本分布匹配损失,将表示对齐到Rectified Generalized Gaussian (RGG)分布。RGG通过整流显式控制期望的$\ell_0$范数,而其连续截断部分在期望$\ell_p$范数和支撑约束下具有最大熵特性。将RDMReg应用于JEPA得到Rectified LpJEPA,它严格推广了先前基于高斯的JEPA。实验表明,Rectified LpJEPA学习到稀疏、非负的表示,具有有利的稀疏性-性能权衡,并在图像分类基准上取得了有竞争力的下游性能,表明RDMReg可以在保留任务相关信息的同时强制执行稀疏性。

英文摘要:

Joint-Embedding Predictive Architectures (JEPA) learn view-invariant representations and admit projection-based distribution matching for collapse prevention. Existing approaches regularize representations towards isotropic Gaussian distributions, but inherently favor dense representations and fail to capture the key property of sparsity observed in efficient representations. We introduce Rectified Distribution Matching Regularization (RDMReg), a sliced two-sample distribution-matching loss that aligns representations to a Rectified Generalized Gaussian (RGG) distribution. RGG enables explicit control over expected $\ell_0$ norm through rectification, while its continuous truncated component admits a maximum-entropy characterization under expected $\ell_p$ norm and support constraints. Equipping JEPAs with RDMReg yields Rectified LpJEPA, which strictly generalizes prior Gaussian-based JEPAs. Empirically, Rectified LpJEPA learns sparse, non-negative representations with favorable sparsity--performance trade-offs and competitive downstream performance on image classification benchmarks, showing that RDMReg can enforce sparsity while preserving task-relevant information.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑