对比学习中使用Sigmoid损失的分析
Analysis of Using Sigmoid Loss for Contrastive Learning
AI总结：
本文从嵌入几何结构角度理论分析对比学习中使用sigmoid损失，提出双常数嵌入模型CCEM，证明其包含sigmoid损失最优嵌入，并揭示最优嵌入随温度参数从单纯形等角紧框架变化到对极结构，合成实验验证了理论结果。
摘要：
对比学习近年来已成为自监督学习的一个重要分支。尤其是CLIP，它将对比学习应用于大规模带字幕图像，获得了广泛关注。最近，CLIP的一种变体SigLIP被提出，它使用sigmoid损失代替标准的InfoNCE损失。SigLIP通过消除对全局视图的需求，以更高效的方式实现了与CLIP相当的性能。然而，在对比学习中使用sigmoid损失的理论理解尚未得到充分探索。本文从学习到的嵌入的几何结构角度，对对比学习中使用sigmoid损失进行了理论分析。首先，我们提出了双常数嵌入模型（CCEM），这是一个通过单一变量参数化各种已知嵌入结构的框架。有趣的是，所提出的CCEM被证明包含关于sigmoid损失的最优嵌入。其次，我们从数学上分析了最小化对比学习sigmoid损失的最优嵌入。最优嵌入根据sigmoid损失中使用的温度参数，从单纯形等角紧框架到对极结构不等。第三，我们在合成数据集上的实验结果与关于最优嵌入结构的理论结果一致。
英文摘要：
Contrastive learning has emerged as a prominent branch of self-supervised learning for several years. Especially, CLIP, which applies contrastive learning to large sets of captioned images, has garnered significant attention. Recently, SigLIP, a variant of CLIP, has been proposed, which uses the sigmoid loss instead of the standard InfoNCE loss. SigLIP achieves the performance comparable to CLIP in a more efficient manner by eliminating the need for a global view. However, theoretical understanding of using the sigmoid loss in contrastive learning is underexplored. In this paper, we provide a theoretical analysis of using the sigmoid loss in contrastive learning, in the perspective of the geometric structure of learned embeddings. First, we propose the double-Constant Embedding Model (CCEM), a framework for parameterizing various well-known embedding structures by a single variable. Interestingly, the proposed CCEM is proven to contain the optimal embedding with respect to the sigmoid loss. Second, we mathematically analyze the optimal embedding minimizing the sigmoid loss for contrastive learning. The optimal embedding ranges from simplex equiangular-tight-frame to antipodal structure, depending on the temperature parameter used in the sigmoid loss. Third, our experimental results on synthetic datasets coincide with the theoretical results on the optimal embedding structures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
