Contrastive learning is a significant paradigm in graph self-supervised learning. However, it requires negative samples to prevent model collapse and learn discriminative representations. These negative samples inevitably lead to heavy computation, memory overhead and class collision, compromising the representation learning. Recent studies present that methods obviating negative samples can attain competitive performance and scalability enhancements, exemplified by bootstrapped graph latents (BGRL). However, BGRL neglects the inherent graph homophily, which provides valuable insights into underlying positive pairs. Our motivation arises from the observation that subtly introducing a few ground-truth positive pairs significantly improves BGRL. Although we can't obtain ground-truth positive pairs without labels under the self-supervised setting, edges in the graph can reflect noisy positive pairs, i.e., neighboring nodes often share the same label. Therefore, we propose to expand the positive pair set with node-neighbor pairs. Subsequently, we introduce a cross-attention module to predict the supportiveness score of a neighbor with respect to the anchor node. This score quantifies the positive support from each neighboring node, and is encoded into the training objective. Consequently, our method mitigates class collision from negative and noisy positive samples, concurrently enhancing intra-class compactness. Extensive experiments are conducted on five benchmark datasets and three downstream task node classification, node clustering, and node similarity search. The results demonstrate that our method generates node representations with enhanced intra-class compactness and achieves state-of-the-art performance.
Self-Labeling Refinement for Robust Representation Learning with Bootstrap Your Own Latent
在这项工作中,我们努力实现两个主要目标。首先,我们研究了批量归一化(BN)层在一个名为Bootstrap Your Own Latent(BYOL)的非对比表示学习框架中的重要性。我们进行了几个实验,得出结论,BN层是不必要的表示学习在BYOL。此外,BYOL只从积极的图像对中学习,而忽略同一批输入中的其他语义相似的图像。对于第二个目标,我们引入了两个新的损失函数来确定同一批输入图像中的语义相似对,并减少它们的表示之间的距离。这些损失函数是交叉余弦相似性损失(CCSL)和交叉S形相似性损失(CSSL)。使用所提出的损失函数,我们能够通过在STL 10数据集上使用CCSL损失(76.87%)训练BYOL框架来超越Vanilla BYOL(71.04%)的性能。使用CSSL loss训练的BYOL使用Vanilla BYOL执行恢复。
Bootstrap your own latent: A new approach to self-supervised Learning
DeepMind;Imperial College(帝国理工学院)
我们介绍Bootstrap Your Own Latent(BYOL),一种新的自监督图像表示学习方法。BYOL依赖于两个神经网络,称为在线网络和目标网络,它们相互作用并相互学习。从图像的增强视图,我们训练在线网络来预测同一图像在不同增强视图下的目标网络表示。同时,我们用在线网络的缓慢移动平均值更新目标网络。虽然最先进的方法依赖于负对,但BYOL在没有负对的情况下达到了新的技术水平。BYOL在ImageNet上使用ResNet-50架构的线性评估和更大ResNet的$79.6\%$达到了$74.3\%$ top-1分类精度。我们表明,BYOL在传输和半监督基准测试中的表现与当前最先进的技术相当或更好。我们的实现和预训练模型在GitHub上提供。
Neighborhood-Adaptive Generalized Linear Graph Embedding with Latent Pattern Mining
School of Computer and Information, Anhui Normal University(安徽师范大学计算机与信息学院);Anhui Provincial Key Laboratory of Industrial Intelligent Data Security(安徽省工业智能数据安全重点实验室);Institute of Automation, State Key Laboratory of Multimodal Artificial Intelligence Systems, Chinese Academy of Sciences(自动化研究所,多模态人工智能系统国家重点实验室,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Learning Structured Twin-Incoherent Twin-Projective Latent Dictionary Pairs for Classification
Soochow University(苏州大学);Hefei University of Technology(合肥工业大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Nanjing University of Information Science and Technology(南京信息工程大学)