arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.03586cs.CVcs.LG
收藏

自监督学习白化损失的研究

An Investigation into Whitening Loss for Self-supervised Learning

Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Beihang University(北京航空航天大学)

Xi Weng, Lei Huang, Lei Zhao, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan

摘要:

自监督学习(SSL)的一个理想目标是避免特征崩溃。白化损失通过在白化来自不同视图的嵌入的条件下最小化正对嵌入之间的距离来保证崩溃避免。在本文中,我们提出了一个框架,一个信息指标来分析白化损失,这提供了一个线索,以解开几个有趣的现象,以及连接到其他SSL方法的枢轴点。我们发现,批白化(BW)为基础的方法不施加白化约束的嵌入,但他们只需要嵌入满秩。这种满秩约束也足以避免维度崩溃。基于我们的分析,我们提出了信道白化与随机组分区(CW-RGP),它利用了BW为基础的方法在防止崩溃的优势,并避免了他们的缺点,需要大批量。ImageNet分类和COCO目标检测的实验结果表明,所提出的CW-RGP具有很好的潜力学习良好的表示。该代码可在https://github.com/winci-ai/CW-RGP上获得。

英文摘要:

A desirable objective in self-supervised learning (SSL) is to avoid feature collapse. Whitening loss guarantees collapse avoidance by minimizing the distance between embeddings of positive pairs under the conditioning that the embeddings from different views are whitened. In this paper, we propose a framework with an informative indicator to analyze whitening loss, which provides a clue to demystify several interesting phenomena as well as a pivoting point connecting to other SSL methods. We reveal that batch whitening (BW) based methods do not impose whitening constraints on the embedding, but they only require the embedding to be full-rank. This full-rank constraint is also sufficient to avoid dimensional collapse. Based on our analysis, we propose channel whitening with random group partition (CW-RGP), which exploits the advantages of BW-based methods in preventing collapse and avoids their disadvantages requiring large batch size. Experimental results on ImageNet classification and COCO object detection reveal that the proposed CW-RGP possesses a promising potential for learning good representations. The code is available at https://github.com/winci-ai/CW-RGP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑