Self-supervised learning based on instance discrimination has shown remarkable progress. In particular, contrastive learning, which regards each image as well as its augmentations as an individual class and tries to distinguish them from all other images, has been verified effective for representation learning. However, pushing away two images that are de facto similar is suboptimal for general representation. In this paper, we propose a hierarchical semantic alignment strategy via expanding the views generated by a single image to \textbf{Cross-samples and Multi-level} representation, and models the invariance to semantically similar images in a hierarchical way. This is achieved by extending the contrastive loss to allow for multiple positives per anchor, and explicitly pulling semantically similar images/patches together at different layers of the network. Our method, termed as CsMl, has the ability to integrate multi-level visual representations across samples in a robust way. CsMl is applicable to current contrastive learning based methods and consistently improves the performance. Notably, using the moco as an instantiation, CsMl achieves a \textbf{76.6\% }top-1 accuracy with linear evaluation using ResNet-50 as backbone, and \textbf{66.7\%} and \textbf{75.1\%} top-1 accuracy with only 1\% and 10\% labels, respectively. \textbf{All these numbers set the new state-of-the-art.}
Texas A&M University（得克萨斯农工大学）；University of Texas at Austin（得克萨斯大学奥斯汀分校）；Google Research, Brain Team（谷歌研究院大脑团队）
Contrastive learning approaches have achieved great success in learning visual representations with few labels of the target classes. That implies a tantalizing possibility of scaling them up beyond a curated "seed" benchmark, to incorporating more unlabeled images from the internet-scale external sources to enhance its performance. However, in practice, larger amount of unlabeled data will require more computing resources due to the bigger model size and longer training needed. Moreover, open-world unlabeled data usually follows an implicit long-tail class or attribute distribution, many of w…
Restoring Vision in Hazy Weather with Hierarchical Contrastive Learning
Nanjing University（南京大学）；Memorial University of Newfoundland（纽芬兰纪念大学）；Australian National University（澳大利亚国立大学）；Sun Yat-sen University（中山大学）；Wenzhou University（温州大学）
Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；ByteDance（字节跳动）；Shanghai Jiao Tong University（上海交通大学）；University of Montréal（蒙特利尔大学）
Hierarchical semantic structures naturally exist in an image dataset, in which several semantically relevant image clusters can be further integrated into a larger cluster with coarser-grained semantics. Capturing such structures with image representations can greatly benefit the semantic understanding on various downstream tasks. Existing contrastive representation learning methods lack such an important model capability. In addition, the negative pairs used in these methods are not guaranteed to be semantically distinct, which could further hamper the structural correctness of learned image …
University of Maryland, College Park（马里兰大学帕克分校）；NVIDIA（英伟达）；Google（谷歌）
One central question for video action recognition is how to model motion. In this paper, we present hierarchical contrastive motion learning, a new self-supervised learning framework to extract effective motion representations from raw video frames. Our approach progressively learns a hierarchy of motion features that correspond to different abstraction levels in a network. This hierarchical design bridges the semantic gap between low-level motion cues and high-level recognition tasks, and promotes the fusion of appearance and motion information at multiple levels. At each level, an explicit m…
Few-shot object detection (FSOD) aims to classify and detect few images of novel categories. Existing meta-learning methods insufficiently exploit features between support and query images owing to structural limitations. We propose a hierarchical attention network with sequentially large receptive fields to fully exploit the query and support images. In addition, meta-learning does not distinguish the categories well because it determines whether the support and query images match. In other words, metric-based learning for classification is ineffective because it does not work directly. Thus,…
Zhejiang University（浙江大学）；JD Explore Academy（京东探索研究院）；National University of Singapore（新加坡国立大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Anomaly detection aims at identifying deviant samples from the normal data distribution. Contrastive learning has provided a successful way to sample representation that enables effective discrimination on anomalies. However, when contaminated with unlabeled abnormal samples in training set under semi-supervised settings, current contrastive-based methods generally 1) ignore the comprehensive relation between training data, leading to suboptimal performance, and 2) require fine-tuning, resulting in low efficiency. To address the above two issues, in this paper, we propose a novel hierarchical …
Balanced Hierarchical Contrastive Learning with Decoupled Queries for Fine-grained Object Detection in Remote Sensing Images
School of Computer Science and Engineering, Nanjing University of Science and Technology, China（南京理工大学计算机科学与工程学院）；College of Computer Science, Zhejiang University, China（浙江大学计算机学院）
Cross-domain EEG-based Emotion Recognition with Contrastive Learning
School of Software Engineering, Xi'an Jiaotong University, China（西安交通大学软件工程学院）；School of Computer Science and Technology, Xi'an Jiaotong University, China（西安交通大学计算机科学与技术学院）
Shanghai Jiaotong University（上海交通大学）；CloudWalk Technology Co., Ltd（云从科技有限公司）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）
The self-supervised ultrasound (US) video model pretraining can use a small amount of labeled data to achieve one of the most promising results on US diagnosis. However, it does not take full advantage of multi-level knowledge for learning deep neural networks (DNNs), and thus is difficult to learn transferable feature representations. This work proposes a hierarchical contrastive learning (HiCo) method to improve the transferability for the US video model pretraining. HiCo introduces both peer-level semantic alignment and cross-level semantic alignment to facilitate the interaction between di…
Current contrastive learning frameworks focus on leveraging a single supervisory signal to learn representations, which limits the efficacy on unseen data and downstream tasks. In this paper, we present a hierarchical multi-label representation learning framework that can leverage all available labels and preserve the hierarchical relationship between classes. We introduce novel hierarchy preserving losses, which jointly apply a hierarchical penalty to the contrastive loss, and enforce the hierarchy constraint. The loss function is data driven and automatically adapts to arbitrary multi-label …
Prior-guided Hierarchical Instance-pixel Contrastive Learning for Ultrasound Speckle Noise Suppression
School of Biological Science and Medical Engineering, Southeast University（生物科学与医学工程学院，东南大学）；Department of Biomedical Engineering, The Ohio State University（生物医学工程系，奥斯特大学）；Jiangsu Key Laboratory of Biomaterials and Devices, Southeast University（江苏省生物材料与器件重点实验室，东南大学）；State Key Laboratory of Digital Medical Engineering, Southeast University（国家数字医学工程重点实验室，东南大学）
A data augmentation module is utilized in contrastive learning to transform the given data example into two views, which is considered essential and irreplaceable. However, the predetermined composition of multiple data augmentations brings two drawbacks. First, the artificial choice of augmentation types brings specific representational invariances to the model, which have different degrees of positive and negative effects on different downstream tasks. Treating each type of augmentation equally during training makes the model learn non-optimal representations for various downstream tasks and…
TreeGaussian: Tree-Guided Cascaded Contrastive Learning for Hierarchical Consistent 3D Gaussian Scene Segmentation and Understanding
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beihang University（北京航空航天大学）
H3Former: Hypergraph-based Semantic-Aware Aggregation via Hyperbolic Hierarchical Contrastive Loss for Fine-Grained Visual Classification
College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；College of Software, Jilin University（吉林大学软件学院）；School of Software, Tsinghua University（清华大学软件学院）
SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptome
Wellcome Sanger Institute（沃森桑格研究所）；Cambridge Centre for AI in Medicine（剑桥人工智能医学中心）；Institute of AI for Health（人工智能与健康研究所）；Cambridge Stem Cell Institute（剑桥干细胞研究所）
Technical University Munich（慕尼黑工业大学）；Helmholtz AI（亥姆霍兹人工智能研究院）
In computational pathology, we often face a scarcity of annotations and a large amount of unlabeled data. One method for dealing with this is semi-supervised learning which is commonly split into a self-supervised pretext task and a subsequent model fine-tuning. Here, we compress this two-stage training into one by introducing S5CL, a unified framework for fully-supervised, self-supervised, and semi-supervised learning. With three contrastive losses defined for labeled, unlabeled, and pseudo-labeled images, S5CL can learn feature representations that reflect the hierarchy of distance relations…
Shaanxi University of Science and Technology（陕西科技大学）；Northwestern Polytechnical University（西北工业大学）；The University of Sydney（悉尼大学）；Shaanxi Normal University（陕西师范大学）；Brunel University London（伦敦布鲁内尔大学）；Tongji University（同济大学）
Morphological reconstruction (MR) is often employed by seeded image segmentation algorithms such as watershed transform and power watershed as it is able to filter seeds (regional minima) to reduce over-segmentation. However, MR might mistakenly filter meaningful seeds that are required for generating accurate segmentation and it is also sensitive to the scale because a single-scale structuring element is employed. In this paper, a novel adaptive morphological reconstruction (AMR) operation is proposed that has three advantages. Firstly, AMR can adaptively filter useless seeds while preserving…
Hierarchical Consensus Network for Multiview Feature Learning
School of Information Science and Engineering, Lanzhou University（兰州大学信息科学与工程学院）；Australian Artificial Intelligence Institute, University of Technology Sydney（悉尼科技大学澳大利亚人工智能学院）
HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Models
Zhejiang University（浙江大学）；Byte Dance（字节跳动）；National University of Singapore（新加坡国立大学）；Angelalign Inc China（Angelalign中国公司）；Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室）
Huazhong University of Science and Technology（华中科技大学）；National University of Singapore（新加坡国立大学）；Alibaba Group（阿里巴巴集团）
Natural videos provide rich visual contents for self-supervised learning. Yet most existing approaches for learning spatio-temporal representations rely on manually trimmed videos, leading to limited diversity in visual patterns and limited performance gain. In this work, we aim to learn representations by leveraging more abundant information in untrimmed videos. To this end, we propose to learn a hierarchy of consistencies in videos, i.e., visual consistency and topical consistency, corresponding respectively to clip pairs that tend to be visually similar when separated by a short time span a…
Emergent Hierarchical Monosemantic Neurons from the Group-Contrastive Forward-Forward Algorithm
National University of Singapore（新加坡国立大学）；Lund University（隆德大学）；Chinese University of Hong Kong（香港中文大学）；Indian Institute of Technology, Dhanbad（印度理工学院（丹巴德分校））
Text-Video Retrieval plays an important role in multi-modal understanding and has attracted increasing attention in recent years. Most existing methods focus on constructing contrastive pairs between whole videos and complete caption sentences, while overlooking fine-grained cross-modal relationships, e.g., clip-phrase or frame-word. In this paper, we propose a novel method, named Hierarchical Cross-Modal Interaction (HCMI), to explore multi-level cross-modal relationships among video-sentence, clip-phrase, and frame-word for text-video retrieval. Considering intrinsic semantic frame relations…
BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning
The Ohio State University（俄亥俄州立大学）；Smithsonian Institution（史密森学会）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）；University of California, Irvine（加州大学伊市分校）；Princeton University（普林斯顿大学）；Duke University（杜克大学）
HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Department of Automation, Tsinghua University（清华大学自动化系）；Beijing National Research Center for Information Science and Technology, Tsinghua University（清华大学北京信息科学与技术国家研究中心）
