PEIRA: 通过视图回归对齐学习预测编码器
PEIRA: Learning Predictive Encoders through Inter-View Regressor Alignment
AI总结：
本文提出PEIRA方法，通过显式目标函数和线性回归器对齐来实现非对比自监督学习，通过理论分析和实验验证其在ImageNet-1K和CIFAR-10上的有效性。
摘要：
非对比自监督学习（SSL）是预测表示学习的有效框架，但像SimSiam、BYOL、I-JEPA或DINO等流行方法依赖于自蒸馏来训练教师-学生网络，但通常不最小化明确的目标函数。我们分析了联合嵌入预测架构（JEPA）的一个变种，使用正则化的线性回归器来预测数据两个视图之间的学习表示，并完全表征其稳定性：非坍塌的稳定平衡点对齐于主导的非线性典型相关子空间，而坍塌的平衡点也可能是稳定的吸引子。受此结果启发，我们引入PEIRA，一种非对比SSL方法，其目标函数通过最优线性回归器的迹定义。我们证明其唯一稳定的平衡点是非平凡的全局最小值，并恢复相同的典型相关子空间，正则化选择有效维度。在ImageNet-1K和CIFAR-10上的实验表明，PEIRA与VICReg和LeJEPA基线具有竞争力，定性实验结果支持理论。
英文摘要：
Non-contrastive self-supervised learning (SSL) is an effective framework for predictive representation learning, but popular (and in practice effective) methods such as SimSiam, BYOL, I-JEPA or DINO, which rely on a form of self-distillation to train a teacher-student network, remain poorly understood as they typically do not minimize a well-defined objective. We analyze the dynamics of a variant of the Joint Embedding Predictive Architecture (JEPA) using a regularized linear regressor to predict the learned representations of two views of the data from one another, and fully characterize its stability: non-collapsed stable equilibria align with leading nonlinear canonical correlation subspaces, while collapsed equilibria may also be stable attractors. Motivated by this result, we introduce PEIRA, a non-contrastive SSL method with an explicit objective defined through the trace of the optimal linear regressor. We show that its only stable equilibria are nontrivial global minimizers and recover the same canonical correlation subspaces, with regularization selecting the effective dimension. Experiments on ImageNet-1K and CIFAR-10 show PEIRA is competitive with VICReg and LeJEPA baselines, and qualitative empirical results support the theory.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
