In this work, we present Multi-Level Contrastive Learning for Dense Prediction Task (MCL), an efficient self-supervised method for learning region-level feature representation for dense prediction tasks. Our method is motivated by the three key factors in detection: localization, scale consistency and recognition. To explicitly encode absolute position and scale information, we propose a novel pretext task that assembles multi-scale images in a montage manner to mimic multi-object scenarios. Unlike the existing image-level self-supervised methods, our method constructs a multi-level contrastive loss that considers each sub-region of the montage image as a singleton. Our method enables the neural network to learn regional semantic representations for translation and scale consistency while reducing pre-training epochs to the same as supervised pre-training. Extensive experiments demonstrate that MCL consistently outperforms the recent state-of-the-art methods on various datasets with significant margins. In particular, MCL obtains 42.5 AP$^\mathrm{bb}$ and 38.3 AP$^\mathrm{mk}$ on COCO with the 1x schedule fintuning, when using Mask R-CNN with R50-FPN backbone pre-trained with 100 epochs. In comparison to MoCo, our method surpasses their performance by 4.0 AP$^\mathrm{bb}$ and 3.1 AP$^\mathrm{mk}$. Furthermore, we explore the alignment between pretext task and downstream tasks. We extend our pretext task to supervised pre-training, which achieves a similar performance to self-supervised learning. This result demonstrates the importance of the alignment between pretext task and downstream tasks, indicating the potential for wider applicability of our method beyond self-supervised settings.
Homeomorphism Prior for False Positive and Negative Problem in Medical Image Dense Contrastive Representation Learning
Southeast University(东南大学);Centre de Recherche en Information Biomédicale Sino-Français (CRIBs)(中法生物医学信息研究中心(CRIBs));Jiangsu Provincial Joint International Research Laboratory of Medical Information Processing(江苏省医疗信息处理联合国际研究实验室);School of Instrument Science and Engineering, Southeast University(东南大学仪器科学与工程学院);Western University(西安大略大学);Case Western Reserve University(凯斯西储大学)
Focus Your Distribution: Coarse-to-Fine Non-Contrastive Learning for Anomaly Detection and Localization
SenseTime Research(商汤科技研究院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);University of California, Los Angeles(加州大学洛杉矶分校)
Unsupervised Dense Information Retrieval with Contrastive Learning
Meta AI Research(Meta人工智能研究院);Ecole normale supérieure, PSL University(巴黎文理研究大学高等师范学院);Inria(法国国家信息与自动化研究所);Université Grenoble Alpes(格勒诺布尔阿尔卑斯大学);University College London(伦敦大学学院)
Spatial Transcriptomics Expression Prediction from Histopathology Based on Cross-Modal Mask Reconstruction and Contrastive Learning
University of Regensburg(雷根斯堡大学);Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)(埃尔朗根-纽伦堡弗里德里希-亚历山大大学);Beijing Friendship Hospital, Capital Medical University(首都医科大学附属北京友谊医院);Hanover Medical School(汉诺威医学院);University Clinic Freiburg(弗莱堡大学附属医院);Fraunhofer Institute for Digital Medicine MEVIS(弗劳恩霍夫数字医学研究所)
From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression
Alibaba(阿里巴巴);Peking University(北京大学)
预训练语言模型(Pre-trained Language Models,PLM)在预训练和微调的范式下在各种自然语言处理(Natural Language Processing,NLP)任务中取得了巨大的成功。PLM具有大量的参数,计算密集型和资源消耗型。因此,模型修剪已被引入到压缩大规模PLM。然而,大多数已有的方法在剪枝时只考虑任务相关的知识,而忽略了任务无关的基本知识,这可能会导致灾难性遗忘问题,从而导致泛化能力差。为了在我们的修剪模型中保持任务不可知和任务特定的知识,我们提出了预训练和微调范式下的主动修剪(CAP)。它被设计为一个通用框架,兼容结构化和非结构化修剪。CAP与对比学习相结合,使修剪后的模型能够从预训练的模型中学习任务不可知的知识,并对任务特定的知识进行微调。此外,为了更好地保持修剪模型的性能,快照(即,在每次修剪迭代的中间模型)也用作修剪的有效监督。我们广泛的实验表明,采用CAP一贯产生显着的改善,特别是在极高的稀疏性的情况下。仅保留3%的模型参数(即,97%稀疏性),CAP成功地实现了99.2%和96.3%的原始BERT性能在QQP和MNLI任务。此外,我们的探测实验表明,CAP修剪的模型倾向于实现更好的泛化能力。
A Self-supervised Contrastive Learning Method for Grasp Outcomes Prediction
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳市人工智能与机器人研究院)
Repetitive Contrastive Learning Enhances Mamba's Selectivity in Time Series Prediction
School of Intelligence Science and Technology, Peking University, Beijing(智能科学与技术学院,北京大学,北京);Computational Intelligence Laboratory(计算智能实验室);Institute for Artificial Intelligence(人工智能研究院);State Key Laboratory of General Artificial Intelligence, Peking University, Beijing(通用人工智能国家重点实验室,北京大学,北京)
Local-Global Multimodal Contrastive Learning for Molecular Property Prediction
School of Mathematical Sciences(数学科学学院);University of Electronic Science and Technology of China(电子科技大学);Department of Computer Science and Engineer(计算机科学与工程系);Oakland University(奥克兰大学);Department of Electrical and Computer Engineering(电气与计算机工程系);College of Management Science(管理科学学院);Chengdu University of Technology(成都理工大学)
Uncertainty-Aware Metabolic Stability Prediction with Dual-View Contrastive Learning
School of Cyber Science and Engineering, Huazhong University of Science and Technology(华中科技大学计算机科学与工程学院);School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件学院);School of Information and Communication Technology, Griffith University(格里菲斯大学信息与通信技术学院);Union Hospital, Tongji Medical College, Huazhong University of Science and Technology(华中科技大学同济医学院附属同济医院)
Autonomous End-to-End SOH Prediction Services for Battery Systems via Temporal-Contrastive Representation Learning
School of Software Engineering, Sun Yat-sen University(中山大学软件学院);Tianneng Battery Group Co., Ltd(天能电池集团有限公司);School of Communication Engineering, Hangzhou Dianzi University(杭州电子科技大学通信工程学院);Institute of Engineering Science, Universidad de O’Higgins(奥希金斯大学工程科学研究所)
CSP-AIT-Net: A contrastive learning-enhanced spatiotemporal graph attention framework for short-term metro OD flow prediction with asynchronous inflow tracking
CoDCL: Counterfactual-Inspired Augmentation Contrastive Learning for Temporal Link Prediction in Social Networks
Jiangsu Key Laboratory of Net-Worked Collective Intelligence, School of Cyber Science and Engineering, Southeast University(江苏网络集体智能重点实验室,网络安全科学与工程学院,东南大学);School of Mathematics, Southeast University(数学学院,东南大学);Frontiers Science Center for Mobile Information Communication and Security, School of Mathematics, Southeast University(移动信息通信与安全前沿科学中心,数学学院,东南大学);Purple Mountain Laboratories(紫金山实验室)
许多对比表示学习方法学习整个图像的单个全局表示。然而,密集对比表示学习方法,如DenseCL(Wang et al.,2021)可以学习更好的表示,用于需要更强的空间定位功能的任务,如多标签分类,检测和分割。在这项工作中,我们研究了如何通过修改训练方案和目标函数来提高DenseCL学习的表示质量,并提出了DenseCL++。我们还进行了几项消融研究,以更好地了解以下因素的影响:(i)在不同图像的增强之间形成密集负对的各种技术,(ii)交叉视图密集负对和正对,以及(iii)辅助重建任务。我们的结果显示,与Simplified相比,mAP改善了3.5%和4%(Chen等人,2020a)和COCO多标签分类中的DenseCL。在COCO和VOC分割任务中,我们分别比Simplified实现了1.8%和0.7%的mIoU改进。
Beijing University of Posts and Telecommunications(北京邮电大学);Rutgers University(罗格斯大学);Shanghai AI Laboratory(上海人工智能实验室);Hasso Plattner Institute, University of Potsdam(波茨坦大学哈索普拉特纳研究所);YITU Technology(依图科技)