arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.03530cs.LGcs.AI
收藏

方向性神经坍塌解释自监督学习中的少样本迁移

Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learning

Department of Computer Science \& Engineering, Texas A\&M University

Achleshwar Luthra, Yash Salunkhe, Tomer Galanti

AI总结:

方向性神经坍塌解释自监督学习中少样本迁移的机制,通过分析决策轴方差与多任务几何的关系,揭示了表示学习中的关键特性。

摘要:

冻结的自监督表示通常能够仅用少量标签在许多语义任务中实现良好的迁移。我们主张一个单一的几何量,即方向性CDNV(决策轴方差),是两种有利行为的核心:在任务内强大的少样本迁移,以及在许多任务中低干扰。我们展示当沿类别分离方向的变异性较小时,这两种现象都会出现。首先,我们证明了下游分类的严格非渐近多类泛化界限,其主导项是方向性CDNV。这些界限包括有限样本修正,清晰地将内在决策轴变异性与质心估计误差分开。其次,我们将决策轴坍塌与多任务几何联系起来:对于独立平衡的标签设置,任务间小的方向性CDNV迫使相应的决策轴几乎正交,帮助单一表示支持许多任务而干扰最小。经验上,在SSL目标中,预训练期间方向性CDNV坍塌,即使经典CDNV仍较大,我们的界限在实际样本大小下紧密跟踪少样本误差。此外,在合成多任务数据上,我们验证SSL学习的表示诱导的决策轴几乎正交。论文的代码和项目页面可在[\href{https://dlfundamentals.github.io/directional-neural-collapse/}{项目页面}]获得。

英文摘要:

Frozen self-supervised representations often transfer well with only a few labels across many semantic tasks. We argue that a single geometric quantity, \emph{directional} CDNV (decision-axis variance), sits at the core of two favorable behaviors: strong few-shot transfer within a task, and low interference across many tasks. We show that both emerge when variability \emph{along} class-separating directions is small. First, we prove sharp non-asymptotic multiclass generalization bounds for downstream classification whose leading term is the directional CDNV. The bounds include finite-shot corrections that cleanly separate intrinsic decision-axis variability from centroid-estimation error. Second, we link decision-axis collapse to multitask geometry: for independent balanced labelings, small directional CDNV across tasks forces the corresponding decision axes to be nearly orthogonal, helping a single representation support many tasks with minimal interference. Empirically, across SSL objectives, directional CDNV collapses during pretraining even when classical CDNV remains large, and our bounds closely track few-shot error at practical shot sizes. Additionally, on synthetic multitask data, we verify that SSL learns representations whose induced decision axes are nearly orthogonal. The code and project page of the paper are available at [\href{https://dlfundamentals.github.io/directional-neural-collapse/}{project page}].

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑