对监督对比损失的神经坍缩几何结构进行工程化设计
Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss
AI总结：
本研究针对监督对比损失的特征嵌入几何结构调控问题，提出通过加入原型调整几何结构的方法，揭示了原型数量远超批次大小时与带固定分类器和归一化嵌入的交叉熵损失的关联，并通过视觉数据集实验验证了结论。
摘要：
监督对比损失（Supervised-contrastive loss，SCL）是分类任务中交叉熵（cross-entropy，CE）的替代方案，它利用嵌入空间中的相似度来生成更丰富的表征。在本研究中，我们提出了通过修改对比损失来对这些学习到的特征嵌入的几何结构进行工程化设计的方法。为调整几何结构，我们探究了原型（prototypes，即训练过程中加入的固定嵌入，用于改变最终的特征几何结构）的影响。具体而言，通过实证结果我们证明，在每个批次中加入原型会促使学习到的嵌入的几何结构与原型的几何结构对齐。我们通过考虑原型数量远超过原始批次大小的极限场景获得了进一步的见解，借此建立了该方法与采用固定分类器和归一化嵌入的交叉熵（CE）损失之间的联系。我们通过在基准视觉数据集上用深度神经网络开展一系列实验验证了我们的研究结果。
英文摘要：
Supervised-contrastive loss (SCL) is an alternative to cross-entropy (CE) for classification tasks that makes use of similarities in the embedding space to allow for richer representations. In this work, we propose methods to engineer the geometry of these learnt feature embeddings by modifying the contrastive loss. In pursuit of adjusting the geometry we explore the impact of prototypes, fixed embeddings included during training to alter the final feature geometry. Specifically, through empirical findings, we demonstrate that the inclusion of prototypes in every batch induces the geometry of the learnt embeddings to align with that of the prototypes. We gain further insights by considering a limiting scenario where the number of prototypes far outnumber the original batch size. Through this, we establish a connection to cross-entropy (CE) loss with a fixed classifier and normalized embeddings. We validate our findings by conducting a series of experiments with deep neural networks on benchmark vision datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
