对比损失的小批量优化
Mini-Batch Optimization of Contrastive Loss
AI总结：
本文研究对比学习的小批量优化理论，证明其与全批量优化等价的条件，并提出基于谱聚类识别高损失小批量以加速 SGD 的方法。
摘要：
对比学习作为一种自监督学习方法已受到广泛关注。对比损失函数确保正样本对（例如来自同一类别的不同样本或同一对象的不同视图）的嵌入彼此相似，而负样本对的嵌入彼此不相似。诸如巨大内存需求之类的实际限制，使得考虑所有可能的正样本对和负样本对具有挑战性，从而导致使用小批量优化。在本文中，我们研究对比学习中小批量优化的理论方面。我们证明，当且仅当所有 $\inom{N}{B}$ 个小批量都被选择时，小批量优化才等价于全批量优化，而当仅考察一个子集时，可能会出现次优性。随后，我们证明利用高损失小批量可以加速 SGD 收敛，并提出一种基于谱聚类的方法来识别这些高损失小批量。我们的实验结果验证了理论发现，并表明所提出的算法在实际相关设置中优于普通 SGD，从而为对比学习中的小批量优化提供了更好的理解。
英文摘要：
Contrastive learning has gained significant attention as a method for self-supervised learning. The contrastive loss function ensures that embeddings of positive sample pairs (e.g., different samples from the same class or different views of the same object) are similar, while embeddings of negative pairs are dissimilar. Practical constraints such as large memory requirements make it challenging to consider all possible positive and negative pairs, leading to the use of mini-batch optimization. In this paper, we investigate the theoretical aspects of mini-batch optimization in contrastive learning. We show that mini-batch optimization is equivalent to full-batch optimization if and only if all $\binom{N}{B}$ mini-batches are selected, while sub-optimality may arise when examining only a subset. We then demonstrate that utilizing high-loss mini-batches can speed up SGD convergence and propose a spectral clustering-based approach for identifying these high-loss mini-batches. Our experimental results validate our theoretical findings and demonstrate that our proposed algorithm outperforms vanilla SGD in practically relevant settings, providing a better understanding of mini-batch optimization in contrastive learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
