arXiv 2209.079022022-09cs.LGcs.CV

MetaMask：重新审视自监督学习中的维度混淆因素

MetaMask: Revisiting Dimensional Confounder for Self-Supervised Learning

University of Chinese Academy of Sciences（中国科学院大学）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；Southern Marine Science and Engineering Guangdong Laboratory (Guangzhou)（南方海洋科学与工程广东省实验室（广州））；Gaoling School of Artificial Intelligence（高瓴人工智能学院）；Renmin University of China（中国人民大学）；Beijing Key Laboratory of Big Data Management and Analysis Methods（北京市大数据管理与分析方法重点实验室）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Guangzhou HKUST Fok Ying Tung Research Institute（广州香港科技大学霍英东研究院）

Jiangmeng Li, Wenwen Qiang, Yanan Zhang, Wenyi Mo, Changwen Zheng, Bing Su, Hui Xiong

备注：: Accepted by NeurIPS 2022 as Spotlight

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 MetaMask 方法，通过元学习训练维度掩码并结合冗余减少技术，解决自监督对比学习中的维度冗余与维度混淆因素问题，从而提升下游分类性能并在多个基准上达到最优效果。

摘要：

作为自监督学习的一种成功方法，对比学习旨在学习输入样本不同失真之间共享的不变信息。尽管对比学习在采样策略和架构设计上不断取得进展，它仍然存在两个持续性的缺陷：任务无关信息的干扰和样本效率低下，这些问题与平凡常数解的反复出现有关。从维度分析的角度出发，我们发现维度冗余和维度混淆因素是这些现象背后的内在问题，并提供了实验证据来支持我们的观点。我们进一步提出了一种简单而有效的方法 MetaMask，即通过元学习学到的维度掩码（dimensional Mask learned by Meta-learning）的缩写，用于学习对维度冗余和混淆因素具有鲁棒性的表示。MetaMask 采用冗余减少技术来解决维度冗余问题，并创新性地引入一个维度掩码来降低包含混淆因素的特定维度的梯度效应，该掩码通过采用元学习范式进行训练，其目标是提高被掩码表示在典型自监督任务上的性能。我们提供了扎实的理论分析，证明与典型的对比学习方法相比，MetaMask 能够为下游分类获得更紧的风险界。在实验上，我们的方法在多个基准上取得了最先进的性能。

英文摘要：

As a successful approach to self-supervised learning, contrastive learning aims to learn invariant information shared among distortions of the input sample. While contrastive learning has yielded continuous advancements in sampling strategy and architecture design, it still remains two persistent defects: the interference of task-irrelevant information and sample inefficiency, which are related to the recurring existence of trivial constant solutions. From the perspective of dimensional analysis, we find out that the dimensional redundancy and dimensional confounder are the intrinsic issues behind the phenomena, and provide experimental evidence to support our viewpoint. We further propose a simple yet effective approach MetaMask, short for the dimensional Mask learned by Meta-learning, to learn representations against dimensional redundancy and confounder. MetaMask adopts the redundancy-reduction technique to tackle the dimensional redundancy issue and innovatively introduces a dimensional mask to reduce the gradient effects of specific dimensions containing the confounder, which is trained by employing a meta-learning paradigm with the objective of improving the performance of masked representations on a typical self-supervised task. We provide solid theoretical analyses to prove MetaMask can obtain tighter risk bounds for downstream classification compared to typical contrastive methods. Empirically, our method achieves state-of-the-art performance on various benchmarks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。