Do Generated Data Always Help Contrastive Learning?
Peking University(北京大学);MIT(麻省理工学院);Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(西安交通大学人工智能与机器人研究所);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);School of Mathematical Sciences, Peking University(北京大学数学科学学院)
Contrastive Learning (CL) has emerged as one of the most successful paradigms for unsupervised visual representation learning, yet it often depends on intensive manual data augmentations. With the rise of generative models, especially diffusion models, the ability to generate realistic images close to the real data distribution has been well recognized. These generated high-equality images have been successfully applied to enhance contrastive representation learning, a technique termed ``data inflation''. However, we find that the generated data (even from a good diffusion model like DDPM) may sometimes even harm contrastive learning. We investigate the causes behind this failure from the perspective of both data inflation and data augmentation. For the first time, we reveal the complementary roles that stronger data inflation should be accompanied by weaker augmentations, and vice versa. We also provide rigorous theoretical explanations for these phenomena via deriving its generalization bounds under data inflation. Drawing from these insights, we propose Adaptive Inflation (AdaInf), a purely data-centric strategy without introducing any extra computation cost. On benchmark datasets, AdaInf can bring significant improvements for various contrastive learning methods. Notably, without using external data, AdaInf obtains 94.70% linear accuracy on CIFAR-10 with SimCLR, setting a new record that surpasses many sophisticated methods. Code is available at https://github.com/PKU-ML/adainf.
Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget
Faculty of Computer Science, University of Vienna, Vienna, Austria(维也纳大学计算机科学学院);UniVie Doctoral School Computer Science, University of Vienna(维也纳大学计算机科学博士学院);Institute of Advanced Research in Artificial Intelligence (IARAI)(高级人工智能研究所)
提出 MAE-CT 方法,结合 MIM 的效率与 ID 的对象聚焦能力,通过 NNCLR 目标在预训练 MAE 顶层诱导语义聚类,无需标签且计算开销低,在 ViT 模型上多项指标达 SOTA。
Perfect Alignment May be Poisonous to Graph Contrastive Learning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Key Laboratory of Big Data Management and Analysis Methods(北京大数据管理与分析方法重点实验室)
ConsPrompt: Exploiting Contrastive Samples for Fewshot Prompt Learning
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Beijing Institute of Technology(北京理工大学);Southeast Academy of Information Technology(东南信息技术研究院)
Uncovering Capabilities of Model Pruning in Graph Contrastive Learning
State Key Laboratory of Complex & Critical Software Environment, Beihang University(北京航空航天大学复杂与关键软件环境国家重点实验室);School of Statistics and Mathematics, Central University of Finance and Economics(中央财经大学统计与数学学院)
Region Embedding with Intra and Inter-View Contrastive Learning
Nanyang Technological University(南洋理工大学)
无监督区域表示学习旨在从未标记的城市数据中提取密集有效的特征。虽然已经基于多个视图做出了一些努力来解决这个问题,但是现有方法在提取视图中的表示和/或合并来自不同视图的表示方面仍然不足。基于对比学习在表示学习中的成功,我们提出将其用于多视图区域表示学习,并设计了一个称为ReMVC(Region Embedding with Multi-View对比学习)的模型,该模型遵循两个指导原则:i)在每个视图中将区域与其他区域进行比较,以实现有效的表征提取; ii)在不同视图中将区域与自身进行比较,以实现跨视图信息共享。我们设计了视图内对比学习模块和视图间对比学习模块,前者用于学习不同的区域嵌入,后者作为软共正则化器来约束嵌入参数并在多视图间传递知识。我们在两个下游任务中利用学习到的区域嵌入,即土地使用聚类和区域流行度预测。大量的实验表明,我们的模型实现了令人印象深刻的改进相比,七个国家的最先进的基线方法,在土地利用聚类任务的利润率超过30%。
Conditional Neural Processes~(CNPs) bridge neural networks with probabilistic inference to approximate functions of Stochastic Processes under meta-learning settings. Given a batch of non-{ \it i.i.d} function instantiations, CNPs are jointly optimized for in-instantiation observation prediction and cross-instantiation meta-representation adaptation within a generative reconstruction pipeline. There can be a challenge in tying together such two targets when the distribution of function observations scales to high-dimensional and noisy spaces. Instead, noise contrastive estimation might be able…
A Fused Gromov-Wasserstein Approach to Subgraph Contrastive Learning
LTCI, Télécom Paris Institut Polytechnique de Paris(巴黎理工学院电信高等学院LTCI实验室);École des Mines de Saint-Étienne(圣埃蒂安矿业学院);Université Paris Dauphine-PSL(巴黎多芬大学-PSL大学);Université Paris-Saclay CentraleSupélec(巴黎萨克雷大学中央高等电力学院);Inria(法国国家信息与自动化研究所)
Dual Perspectives on Non-Contrastive Self-Supervised Learning
Meta FAIR;New York University(纽约大学);INRIA Paris(法国国家信息与自动化研究所巴黎中心);Carnegie-Mellon University(卡内基梅隆大学);Univ. Grenoble Alpes(格勒诺布尔阿尔卑斯大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);Grenoble INP(格勒诺布尔国立高等物理、数学与信息学院)
Contrastive Learning Via Equivariant Representation
Xi’an Jiaotong-Liverpool University(西交利物浦大学);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);University of Science and Technology of China(中国科学技术大学);University of Liverpool(利物浦大学)
Beijing National Research Center for Information Science and Technology(BNRist)(北京信息科学与技术国家研究中心);Department of Computer Science and Technology Tsinghua University(清华大学计算机科学与技术系)
Repetitive Contrastive Learning Enhances Mamba's Selectivity in Time Series Prediction
School of Intelligence Science and Technology, Peking University, Beijing(智能科学与技术学院,北京大学,北京);Computational Intelligence Laboratory(计算智能实验室);Institute for Artificial Intelligence(人工智能研究院);State Key Laboratory of General Artificial Intelligence, Peking University, Beijing(通用人工智能国家重点实验室,北京大学,北京)
Topology Reorganized Graph Contrastive Learning with Mitigating Semantic Drift
College of Computer Science & Technology, Nanjing University of Aeronautics & Astronautics(南京航空航天大学计算机科学与技术学院);MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing University of Aeronautics & Astronautics(南京航空航天大学工信部模式分析与机器智能重点实验室)
MHCCL: Masked Hierarchical Cluster-Wise Contrastive Learning for Multivariate Time Series
Shandong University(山东大学);Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University(山东大学-南洋理工大学联合人工智能研究中心(C-FAIR));Lund University(隆德大学);Nanyang Technological University(南洋理工大学)
School of Life Sciences, Tiangong University(天津工大学生命科学学院);School of Electronics and Information Engineering, Tiangong University(天津工业大学电子与信息工程学院);School of Software, Beihang University(北京航空航天大学软件学院)
HC$^2$L: Hybrid and Cooperative Contrastive Learning for Cross-lingual Spoken Language Understanding
University of Science and Technology Beijing(北京科技大学);Agency for Science, Technology and Research(新加坡科技研究局);Nanyang Technological University(南洋理工大学);University of Technology Sydney(悉尼科技大学)