自监督预训练具有无需人工标注即可生成表达性表征的潜力。地球观测（EO）领域的大多数预训练基于ImageNet或中等规模、带标注的遥感（RS）数据集。我们公开了一个无标注遥感数据集SSL4EO-S12（Self-Supervised Learning for Earth Observation - Sentinel-1/2），用于汇集来自ESA Sentinel-1和Sentinel-2卫星任务的大规模、全球、多模态和多季节卫星影像语料库。针对EO应用，我们证明了SSL4EO-S12能够成功用于一组方法的自监督预训练：MoCo-v2、DINO、MAE和data2vec。所得模型的下游性能接近或超过监督学习的准确率指标。此外，与现有数据集相比，在SSL4EO-S12上进行预训练表现更优。我们在https://github.com/zhu-xlab/SSL4EO-S12上公开提供该数据集、相关源代码和预训练模型。
英文摘要：
Self-supervised pre-training bears potential to generate expressive representations without human annotation. Most pre-training in Earth observation (EO) are based on ImageNet or medium-size, labeled remote sensing (RS) datasets. We share an unlabeled RS dataset SSL4EO-S12 (Self-Supervised Learning for Earth Observation - Sentinel-1/2) to assemble a large-scale, global, multimodal, and multi-seasonal corpus of satellite imagery from the ESA Sentinel-1 \& -2 satellite missions. For EO applications we demonstrate SSL4EO-S12 to succeed in self-supervised pre-training for a set of methods: MoCo-v2, DINO, MAE, and data2vec. Resulting models yield downstream performance close to, or surpassing accuracy measures of supervised learning. In addition, pre-training on SSL4EO-S12 excels compared to existing datasets. We make openly available the dataset, related source code, and pre-trained models at https://github.com/zhu-xlab/SSL4EO-S12.
Human Activity Recognition is a field of research where input data can take many forms. Each of the possible input modalities describes human behaviour in a different way, and each has its own strengths and weaknesses. We explore the hypothesis that leveraging multiple modalities can lead to better recognition. Since manual annotation of input data is expensive and time-consuming, the emphasis is made on self-supervised methods which can learn useful feature representations without any ground truth labels. We extend a number of recent contrastive self-supervised approaches for the task of Huma…
University of Cambridge（剑桥大学）；Huawei R&D UK（华为英国研发中心）
Multi-Modal Self-Supervised Learning from videos has been shown to improve model's performance on various downstream tasks. However, such Self-Supervised pre-training requires large batch sizes and a large amount of computation resources due to the noise present in the uncurated data. This is partly due to the fact that the prevalent training scheme is trained on coarse-grained setting, in which vectors representing the whole video clips or natural language sentences are used for computing similarity. Such scheme makes training noisy as part of the video clips can be totally not correlated wit…
CorMulT: A Semi-supervised Modality Correlation-aware Multimodal Transformer for Sentiment Analysis
Information Networking Institute at Carnegie Mellon University (CMU)（卡内基梅隆大学信息网络研究所）；School of Computer Science and Technology at Tongji University（同济大学计算机科学与技术学院）
Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration
Department of Computer Science, University of Kaiserslautern-Landau (RPTU)（科斯拉尔特伦大学计算机科学系）；SDS, German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；INRAE, UMR TETIS, University of Montpellier（蒙彼利埃大学UMR TETIS）；CIRAD, UMR TETIS, University of Montpellier（蒙彼利埃大学UMR TETIS）；INRIA, EVERGREEN, University of Montpellier（蒙彼利埃大学）
M3LEO: A Multi-Modal, Multi-Label Earth Observation Dataset Integrating Interferometric SAR and Multispectral Data
University of Cambridge（剑桥大学）；Drexel University（德雷塞尔大学）；Universitat de València（瓦伦西亚大学）；European Space Agency（欧洲空间局）；University of Oxford（牛津大学）；Universidad de Antioquia（安蒂奥基亚大学）
Xidian University（西安电子科技大学）；Tencent AI Lab（腾讯人工智能实验室）；University of Sydney（悉尼大学）
Thanks to the success of deep learning, cross-modal retrieval has made significant progress recently. However, there still remains a crucial bottleneck: how to bridge the modality gap to further enhance the retrieval accuracy. In this paper, we propose a self-supervised adversarial hashing (\textbf{SSAH}) approach, which lies among the early attempts to incorporate adversarial learning into cross-modal hashing in a self-supervised fashion. The primary contribution of this work is that two adversarial networks are leveraged to maximize the semantic correlation and consistency of the representat…
Due to abundance of data from multiple modalities, cross-modal retrieval tasks with image-text, audio-image, etc. are gaining increasing importance. Of the different approaches proposed, supervised methods usually give significant improvement over their unsupervised counterparts at the additional cost of labeling or annotation of the training data. Semi-supervised methods are recently becoming popular as they provide an elegant framework to balance the conflicting requirement of labeling cost and accuracy. In this work, we propose a novel deep semi-supervised framework which can seamlessly han…
Nowadays the measure between heterogeneous data is still an open problem for cross-modal retrieval. The core of cross-modal retrieval is how to measure the similarity between different types of data. Many approaches have been developed to solve the problem. As one of the mainstream, approaches based on subspace learning pay attention to learning a common subspace where the similarity among multi-modal data can be measured directly. However, many of the existing approaches only focus on learning a latent subspace. They ignore the full use of discriminative information so that the semantically s…
MIRROR: Multi-Modal Pathological Self-Supervised Representation Learning via Modality Alignment and Retention
The University of Sydney（悉尼大学）；School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（集成空天地海大数据应用技术国家工程实验室）；School of Computer Science and Engineering, Northwestern Polytechnical University（西北工业大学计算机科学与工程学院）；University of Maryland（马里兰大学）；Ningbo Institute of Northwestern Polytechnical University（西北工业大学宁波学院）
Hasso Plattner Institute for Digital Engineering, University of Potsdam（波茨坦大学哈索·普拉特纳数字工程研究所）；TU Kaiserslautern（凯泽斯劳滕工业大学）；Hasso Plattner Institute for Digital Health at the Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院哈索·普拉特纳数字健康研究所）
High annotation costs are a substantial bottleneck in applying modern deep learning architectures to clinically relevant medical use cases, substantiating the need for novel algorithms to learn from unlabeled data. In this work, we propose ContIG, a self-supervised method that can learn from large datasets of unlabeled medical images and genetic data. Our approach aligns images and several genetic modalities in the feature space using a contrastive loss. We design our method to integrate multiple modalities of each individual person in the same model end-to-end, even when the available modalit…
Recently self supervised learning has seen explosive growth and use in variety of machine learning tasks because of its ability to avoid the cost of annotating large-scale datasets. This paper gives an overview for best self supervised learning approaches for multimodal learning. The presented approaches have been aggregated by extensive study of the literature and tackle the application of self supervised learning in different ways. The approaches discussed are cross modal generation, cross modal pretraining, cyclic translation, and generating unimodal labels in self supervised fashion.
Hangzhou Dianzi University（杭州电子科技大学）；Baidu Inc.（百度公司）；Zhejiang International Studies University（浙江外国语学院）
This paper proposes an approach to Dense Video Captioning (DVC) without pairwise event-sentence annotation. First, we adopt the knowledge distilled from relevant and well solved tasks to generate high-quality event proposals. Then we incorporate contrastive loss and cycle-consistency loss typically applied to cross-modal retrieval tasks to build semantic matching between the proposals and sentences, which are eventually used to train the caption generation module. In addition, the parameters of matching module are initialized via pre-training based on annotated images to improve the matching p…
Radboud University Medical Center（拉德堡德大学医学中心）；The Netherlands Cancer Institute（荷兰癌症研究所）；Nanjing University of Information Science and Technology（南京信息工程大学）
Domain adaptation has been widely adopted to transfer styles across multi-vendors and multi-centers, as well as to complement the missing modalities. In this challenge, we proposed an unsupervised domain adaptation framework for cross-modality vestibular schwannoma (VS) and cochlea segmentation and Koos grade prediction. We learn the shared representation from both ceT1 and hrT2 images and recover another modality from the latent representation, and we also utilize proxy tasks of VS segmentation and brain parcellation to restrict the consistency of image structures in domain adaptation. After …
University of British Columbia（不列颠哥伦比亚大学）；Vector Institute for AI（Vector人工智能研究所）
Multi-modal learning, particularly among imaging and linguistic modalities, has made amazing strides in many high-level fundamental visual understanding problems, ranging from language grounding to dense event captioning. However, much of the research has been limited to approaches that either do not take audio corresponding to video into account at all, or those that model the audio-visual correlations in service of sound or sound source localization. In this paper, we present the evidence, that audio signals can carry surprising amount of information when it comes to high-level visual-lingua…
CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillation
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
Weakly-supervised audio-visual violence detection aims to distinguish snippets containing multimodal violence events with video-level labels. Many prior works perform audio-visual integration and interaction in an early or intermediate manner, yet overlooking the modality heterogeneousness over the weakly-supervised setting. In this paper, we analyze the modality asynchrony and undifferentiated instances phenomena of the multiple instance learning (MIL) procedure, and further investigate its negative impact on weakly-supervised audio-visual learning. To address these issues, we propose a modal…
Novo Nordisk Foundation Center for Biosustainability, Technical University of Denmark（丹麦技术大学诺和诺德基金会生物可持续性中心）；University of Copenhagen（哥本哈根大学）
Multimodal generative models should be able to learn a meaningful latent representation that enables a coherent joint generation of all modalities (e.g., images and text). Many applications also require the ability to accurately sample modalities conditioned on observations of a subset of the modalities. Often not all modalities may be observed for all training data points, so semi-supervised learning should be possible. In this study, we propose a novel product-of-experts (PoE) based variational autoencoder that have these desired properties. We benchmark it against a mixture-of-experts (MoE)…
Jiangnan University（江南大学）；University of Surrey（萨里大学）
With the advantage of low storage cost and high retrieval efficiency, hashing techniques have recently been an emerging topic in cross-modal similarity search. As multiple modal data reflect similar semantic content, many researches aim at learning unified binary codes. However, discriminative hashing features learned by these methods are not adequate. This results in lower accuracy and robustness. We propose a novel hashing learning framework which jointly performs classifier learning, subspace learning and matrix factorization to preserve class-specific semantic content, termed Discriminativ…
Complex objects are usually with multiple labels, and can be represented by multiple modal representations, e.g., the complex articles contain text and image information as well as multiple annotations. Previous methods assume that the homogeneous multi-modal data are consistent, while in real applications, the raw data are disordered, e.g., the article constitutes with variable number of inconsistent text and image instances. Therefore, Multi-modal Multi-instance Multi-label (M3) learning provides a framework for handling such task and has exhibited excellent performance. However, M3 learning…
Cross-modal retrieval between visual data and natural language description remains a long-standing challenge in multimedia. While recent image-text retrieval methods offer great promise by learning deep representations aligned across modalities, most of these methods are plagued by the issue of training with small-scale datasets covering a limited number of images with ground-truth sentences. Moreover, it is extremely expensive to create a larger dataset by annotating millions of images with sentences and may lead to a biased model. Inspired by the recent success of webly supervised learning i…
OPPO Research Institute（OPPO研究院）；Xmotors（小鹏汽车）；University of California, San Diego（加州大学圣地亚哥分校）
Most existing human matting algorithms tried to separate pure human-only foreground from the background. In this paper, we propose a Virtual Multi-modality Foreground Matting (VMFM) method to learn human-object interactive foreground (human and objects interacted with him or her) from a raw RGB image. The VMFM method requires no additional inputs, e.g. trimap or known background. We reformulate foreground matting as a self-supervised multi-modality problem: factor each input image into estimated depth map, segmentation mask, and interaction heatmap using three auto-encoders. In order to fully …
Facebook AI（Facebook 人工智能研究院）；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）
Visual and audio modalities are highly correlated, yet they contain different information. Their strong correlation makes it possible to predict the semantics of one from the other with good accuracy. Their intrinsic differences make cross-modal prediction a potentially more rewarding pretext task for self-supervised learning of video and audio representations compared to within-modality learning. Based on this intuition, we propose Cross-Modal Deep Clustering (XDC), a novel self-supervised method that leverages unsupervised clustering in one modality (e.g., audio) as a supervisory signal for …
EarthEmbeddingExplorer: A Web Application for Cross-Modal Retrieval of Global Satellite Images
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences（中国科学院地理科学与资源研究所）；Asterisk Labs（Asterisk实验室）；LGND AI, Inc.（LGND AI公司）；University College London（伦敦大学学院）
The automatic diagnosis of various retinal diseases from fundus images is important to support clinical decision-making. However, developing such automatic solutions is challenging due to the requirement of a large amount of human-annotated data. Recently, unsupervised/self-supervised feature learning techniques receive a lot of attention, as they do not need massive annotations. Most of the current self-supervised methods are analyzed with single imaging modality and there is no method currently utilize multi-modal images for better results. Considering that the diagnostics of various vitreor…
SCPNet: Unsupervised Cross-modal Homography Estimation via Intra-modal Self-supervised Learning
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息与电子工程学院）；Ningbo Innovation Center, Zhejiang University（浙江大学宁波创新中心）；HAOMO.AI Technology Co., Ltd.（毫末智行科技有限公司）；Zhejiang GongShang University（浙江工商大学）
Semi-Supervised Multi-Modal Medical Image Segmentation for Complex Situations
School of Physics, Peking University, Beijing, China（北京大学物理系）；School of Computer Science, Peking University, Beijing, China（北京大学计算机系）；Department of Radiotherapy, Peking University Cancer Hospital, Beijing, China（北京大学肿瘤医院放疗科）
CmFNet: Cross-modal Fusion Network for Weakly-supervised Segmentation of Medical Images
School of Physics, Peking University（北京大学物理学院）；School of Computer Science, Peking University（北京大学计算机学院）；Department of Radiotherapy, Peking University Cancer Hospital（北京大学肿瘤医院放疗科）；Department of Radiation Oncology, Peking University Third Hospital（北京大学第三医院放疗科）
The University of Tokyo（东京大学）；RIKEN AIP（理化学研究所先进智能项目中心）；National University of Defense Technology（国防科技大学）；Hitachi, Ltd（日立制作所）
Self-supervised cross-modal super-resolution (SR) can overcome the difficulty of acquiring paired training data, but is challenging because only low-resolution (LR) source and high-resolution (HR) guide images from different modalities are available. Existing methods utilize pseudo or weak supervision in LR space and thus deliver results that are blurry or not faithful to the source modality. To address this issue, we present a mutual modulation SR (MMSR) model, which tackles the task by a mutual modulation strategy, including a source-to-guide modulation and a guide-to-source modulation. In t…
The City University of New York（纽约市立大学）；Northwestern Polytechnical University（西北工业大学）
Most of the existing self-supervised feature learning methods for 3D data either learn 3D features from point cloud data or from multi-view images. By exploring the inherent multi-modality attributes of 3D objects, in this paper, we propose to jointly learn modal-invariant and view-invariant features from different modalities including image, point cloud, and mesh with heterogeneous networks for 3D data. In order to learn modal- and view-invariant features, we propose two types of constraints: cross-modal invariance constraint and cross-view invariant constraint. Cross-modal invariance constra…
EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain
Advanced Research Institute of Multidisciplinary Sciences, Beijing Institute of Technology（北京理工大学多学科科学先进研究院）；School of Mechatronical Engineering, Beijing Institute of Technology（北京理工大学机电工程学院）；Yangtze Delta Region Academy of Beijing Institute of Technology（北京理工大学长三角研究院）；Beijing Institute of Technology（北京理工大学）
German Aerospace Center（德国航空航天中心）；Technical University of Munich（慕尼黑工业大学）；The University of Tokyo（东京大学）；RIKEN（日本理化学研究所）；Wuhan University（武汉大学）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；INRIA（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）
This paper addresses the problem of semi-supervised transfer learning with limited cross-modality data in remote sensing. A large amount of multi-modal earth observation images, such as multispectral imagery (MSI) or synthetic aperture radar (SAR) data, are openly available on a global scale, enabling parsing global urban scenes through remote sensing imagery. However, their ability in identifying materials (pixel-wise classification) remains limited, due to the noisy collection environment and poor discriminative information as well as limited number of well-annotated training images. To this…
On Uni-Modal Feature Learning in Supervised Multi-Modal Learning
Tsinghua University（清华大学）；University of California, Berkeley（加州大学伯克利分校）；Massachusetts Institute of Technology（麻省理工学院）；Shanghai Qi Zhi Institute（上海期智研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition
Technische Universität Berlin（柏林工业大学）；Science of Intelligence, Research Cluster of Excellence（科学智慧卓越研究集群）；Univ. Grenoble Alpes（格勒诺布尔大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；LJK（让·昆廷诺实验室）；Humboldt-Universität zu Berlin（柏林洪堡大学）
Due to the availability of multi-modal remote sensing (RS) image archives, one of the most important research topics is the development of cross-modal RS image retrieval (CM-RSIR) methods that search semantically similar images across different modalities. Existing CM-RSIR methods require the availability of a high quality and quantity of annotated training images. The collection of a sufficient number of reliable labeled images is time consuming, complex and costly in operational scenarios, and can significantly affect the final accuracy of CM-RSIR. In this paper, we introduce a novel self-su…
Weakly Supervised Visible-Infrared Person Re-Identification via Heterogeneous Expert Collaborative Consistency Learning
Faculty of Information Engineering and Automation, Kunming University of Science and Technology（昆明理工大学信息工程与自动化学院）；School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）
Sun Yat-sen University（中山大学）；Beijing Jiaotong University（北京交通大学）；Alibaba Group（阿里巴巴集团）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Nowadays, customer's demands for E-commerce are more diversified, which introduces more complications to the product retrieval industry. Previous methods are either subject to single-modal input or perform supervised image-level product retrieval, thus fail to accommodate real-life scenarios where enormous weakly annotated multi-modal data are present. In this paper, we investigate a more realistic setting that aims to perform weakly-supervised multi-modal instance-level product retrieval among fine-grained product categories. To promote the study of this challenging task, we contribute Produc…
