Self-supervised learning (SSL) aims to learn intrinsic features without labels. Despite the diverse architectures of SSL methods, the projection head always plays an important role in improving the performance of the downstream task. In this work, we systematically investigate the role of the projection head in SSL. Specifically, the projection head targets the uniformity part of SSL, which pushes the dissimilar samples away from each other, thus enabling the encoder to focus on extracting semantic features. Based on this understanding, we propose a Representation Evaluation Design (RED) in SSL models in which a shortcut connection between the representation and the projection vectors is built. Extensive experiments with different architectures, including SimCLR, MoCo-V2, and SimSiam, on various datasets, demonstrate that the representation evaluation design can consistently improve the baseline models in the downstream tasks. The learned representation from the RED-SSL models shows superior robustness to unseen augmentations and out-of-distribution data.
Projection Head is Secretly an Information Bottleneck
College of Engineering, Peking University(北京大学工学院);School of EECS, Peking University(北京大学电子工程与计算机科学学院);State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院通用人工智能国家重点实验室);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院)
本文从信息论角度揭示对比学习中 projection head 充当信息瓶颈的机制,并基于理论引入训练与结构正则化修改,在多个数据集上实现下游性能的持续提升。
Radial Basis Function Networks as Projection Heads in Self-Supervised Learning
Osnabrück University of Applied Sciences(奥斯纳布吕克应用科学大学);Osnabrück University(奥斯纳布吕克大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心)
Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projection
Wuhan University(武汉大学);Shanghai Innovation Institute(上海创新研究院);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);University of Science and Technology of China(中国科学技术大学);China University of Mining and Technology(中国矿业大学)
Role-Aware Neural Convex Divergence Heads for Asymmetric Representation Learning
School of Mathematics and Statistics, Chongqing Technology and Business University(重庆工商大学数学与统计学院);Chongqing Key Laboratory of Statistical Intelligent Computing and Monitoring, Chongqing Technology and Business University(重庆工商大学统计智能计算与监测重庆市重点实验室);School of Food Science, Chongqing Technology and Business University(重庆工商大学食品科学与工程学院);Faculty of Electrical Engineering and Information Technology, TU Dortmund University(多特蒙德工业大学电气工程与信息技术学院)
Enhancing Data Efficiency and Feature Identification for Lithium-Ion Battery Lifespan Prediction by Deciphering Interpretation of Temporal Patterns and Cyclic Variability Using Attention-Based Models
Korea Advanced Institute of Science and Technology(韩国科学技术院);University of Southern California(南加州大学)
An Empirical Study on Improving SimCLR's Nonlinear Projection Head using Pretrained Autoencoder Embeddings
Faculty of Engineering(工程学院);Computer Science Osnabrück University of Applied Sciences(计算机科学奥斯纳布吕克应用科学大学);Semantic Information Systems Group Osnabrück University(奥斯纳布吕克大学语义信息系统组);German Research Center for AI (DFKI)(德国人工智能研究中心(DFKI))
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China(数学科学国家重点实验室,数学与系统科学研究院,中国科学院,北京100190,中国);School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China(中国科学院大学数学科学学院,北京100049,中国);Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai, China(上海数学与交叉科学研究所(SIMIS),上海,中国);Key Laboratory of Systems Health Science of Zhejiang Province, School of Life Science, Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Chinese Academy of Sciences, Hangzhou 310024, China(浙江省系统健康科学重点实验室,生命科学学院,杭州先进研究院,中国科学院大学,中国科学院,杭州310024,中国)
Scalable Heterogeneous Graph Learning via Heterogeneous-aware Orthogonal Prototype Experts
National Engineering Research Center for Big Data Technology;System Lab, Cluster;Grid Computing Lab, School of Computer Science;Technology, Huazhong University of Science;DIRO, Université de Montréal \& Mila \& Canada CIFAR AI Chair, Canada