arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.12189cs.LG
收藏

解读投影头:表征评价自监督学习

Deciphering the Projection Head: Representation Evaluation Self-supervised Learning

Hongkong University of Science and Technology(香港科技大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)

Jiajun Ma, Tianyang Hu, Wenjia Wang

摘要:

自监督学习(Self-supervised learning,SSL)的目标是学习没有标签的内在特征。尽管SSL方法的架构多种多样,但投影头在提高下游任务的性能方面始终起着重要作用。在这项工作中,我们系统地研究了投影头在SSL中的作用。具体地,投影头针对SSL的均匀性部分,其将不相似的样本彼此推开,从而使编码器能够专注于提取语义特征。基于这种理解,我们提出了一种SSL模型中的表示评估设计(RED),其中建立了表示和投影向量之间的快捷连接。在不同数据集上使用不同架构(包括Simplified、MoCo-V2和SimSiam)进行的大量实验表明,表示评估设计可以持续改进下游任务中的基线模型。来自RED-SSL模型的学习表示显示出对不可见的增强和分布外数据的卓越鲁棒性。

英文摘要:

Self-supervised learning (SSL) aims to learn intrinsic features without labels. Despite the diverse architectures of SSL methods, the projection head always plays an important role in improving the performance of the downstream task. In this work, we systematically investigate the role of the projection head in SSL. Specifically, the projection head targets the uniformity part of SSL, which pushes the dissimilar samples away from each other, thus enabling the encoder to focus on extracting semantic features. Based on this understanding, we propose a Representation Evaluation Design (RED) in SSL models in which a shortcut connection between the representation and the projection vectors is built. Extensive experiments with different architectures, including SimCLR, MoCo-V2, and SimSiam, on various datasets, demonstrate that the representation evaluation design can consistently improve the baseline models in the downstream tasks. The learned representation from the RED-SSL models shows superior robustness to unseen augmentations and out-of-distribution data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑