Both masked image modeling (MIM) and natural language supervision have facilitated the progress of transferable visual pre-training. In this work, we seek the synergy between two paradigms and study the emerging properties when MIM meets natural language supervision. To this end, we present a novel masked visual Reconstruction In Language semantic Space (RILS) pre-training framework, in which sentence representations, encoded by the text encoder, serve as prototypes to transform the vision-only signals into patch-sentence probabilities as semantically meaningful MIM reconstruction targets. The vision models can therefore capture useful components with structured information by predicting proper semantic of masked tokens. Better visual representations could, in turn, improve the text encoder via the image-text alignment objective, which is essential for the effective MIM target transformation. Extensive experimental results demonstrate that our method not only enjoys the best of previous MIM and CLIP but also achieves further improvements on various tasks due to their mutual benefits. RILS exhibits advanced transferability on downstream classification, detection, and segmentation, especially for low-shot regimes. Code will be made available at https://github.com/hustvl/RILS.
MM-3DScene: 3D Scene Understanding by Customizing Masked Modeling with Informative-Preserved Reconstruction and Self-Distilled Consistency
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Shenzhen Institutes of Advanced Technology, Chinese Academy of Science（中国科学院深圳先进技术研究院）；China Telecom Corporation Ltd. Data&AI Technology Company（中国电信股份有限公司数据与AI科技公司）；University of Rochester（罗切斯特大学）；University of Science Technology of China（中国科学技术大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Masked Autoencoders (MAE) have been prevailing paradigms for large-scale vision representation pre-training. By reconstructing masked image patches from a small portion of visible image regions, MAE forces the model to infer semantic correlation within an image. Recently, some approaches apply semantic-rich teacher models to extract image features as the reconstruction target, leading to better performance. However, unlike the low-level features such as pixel values, we argue the features extracted by powerful teacher models already encode rich semantic correlation across regions in an intact …
Efficient Self-supervised Vision Pretraining with Local Masked Reconstruction
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Monash University（莫纳什大学）；Nile University（尼罗河大学）；Nanyang Technological University（南洋理工大学）
Shanghai AI Laboratory（上海人工智能实验室）；Simon Fraser University（西蒙弗雷泽大学）；The Chinese University of Hong Kong（香港中文大学）
Masked Image Modeling (MIM) has achieved promising progress with the advent of Masked Autoencoders (MAE) and BEiT. However, subsequent works have complicated the framework with new auxiliary tasks or extra pre-trained models, inevitably increasing computational overhead. This paper undertakes a fundamental analysis of MIM from the perspective of pixel reconstruction, which examines the input image patches and reconstruction target, and highlights two critical but previously overlooked bottlenecks. Based on this analysis, we propose a remarkably simple and effective method, {\ourmethod}, that e…
Masked Image Modeling (MIM) achieves outstanding success in self-supervised representation learning. Unfortunately, MIM models typically have huge computational burden and slow learning process, which is an inevitable obstacle for their industrial applications. Although the lower layers play the key role in MIM, existing MIM models conduct reconstruction task only at the top layer of encoder. The lower layers are not explicitly guided and the interaction among their patches is only used for calculating new activations. Considering the reconstruction task requires non-trivial inter-patch intera…
Exploring Adaptive Masked Reconstruction for Self-Supervised Skeleton-Based Action Recognition
Hefei University of Technology（合肥工业大学）；Jilin University（吉林大学）；Zhejiang Gongshang University（浙江工商大学）；University of Science and Technology of China（中国科学技术大学）
Masked Registration and Autoencoding of CT Images for Predictive Tibia Reconstruction
Technical University Berlin（技术大学柏林）；Charité – Universitätsmedizin Berlin（柏林查理医院）；Freie Universität Berlin（自由大学柏林）；Humboldt-Universität zu Berlin（洪堡大学柏林）；Institute of Medical Informatics（医学信息学研究所）；Center for Musculoskeletal Surgery（骨科手术中心）；Julius Wolff Institute（朱利叶斯·沃尔夫研究所）
Unsupervised Anomaly Detection via Masked Diffusion Posterior Sampling
School of Automation Engineering, University of Electronic Science and Technology of China(UESTC)（电子科技大学自动化工程学院）；Shenzhen Institute for Advanced Study, UESTC（电子科技大学深圳高等研究院）
The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；Carnegie Mellon University（卡内基梅隆大学）
This paper introduces the Masked Voxel Jigsaw and Reconstruction (MV-JAR) method for LiDAR-based self-supervised pre-training and a carefully designed data-efficient 3D object detection benchmark on the Waymo dataset. Inspired by the scene-voxel-point hierarchy in downstream 3D object detectors, we design masking and reconstruction strategies accounting for voxel distributions in the scene and local point distributions within the voxel. We employ a Reversed-Furthest-Voxel-Sampling strategy to address the uneven distribution of LiDAR points and propose MV-JAR, which combines two techniques for …
Unsupervised Anomaly Detection in Medical Images with a Memory-augmented Multi-level Cross-attentional Masked Autoencoder
Harvard University（哈佛大学）；Singapore Management University（新加坡管理大学）；Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）；South Australian Health and Medical Research Institute（南澳大利亚健康与医学研究所）；University of Surrey（萨里大学）
Pre-training Point Cloud Compact Model with Partial-aware Reconstruction
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Research Center of Artificial Intelligence, Peng Cheng Laboratory（鹏城实验室人工智能研究中心）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）
Computed Tomography (CT) is an imaging technique where information about an object are collected at different angles (called projections or scans). Then the cross-sectional image showing the internal structure of the slice is produced by solving an inverse problem. Limited by certain factors such as radiation dosage, projection angles, the produced images can be noisy or contain artifacts. Inspired by the success of transformer for natural language processing, the core idea of this preliminary study is to consider a projection of tomography as a word token, and the whole scan of the cross-sect…
Wavelet-Driven Masked Image Modeling: A Path to Efficient Visual Representation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Peng Cheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Jiao Tong University（上海交通大学）
MambaMIR: An Arbitrary-Masked Mamba for Joint Medical Image Reconstruction and Uncertainty Estimation
Imperial College London（伦敦帝国学院）；Royal Brompton Hospital（皇家布朗普顿医院）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；University of Cambridge（剑桥大学）
SIDME: Self-supervised Image Demoiréing via Masked Encoder-Decoder Reconstruction
College of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；College of Electronic Information and Electrical Engineering, Shanghai Jiaotong University（上海交通大学电子信息与电气工程学院）；College of Information, Shanghai Ocean University（上海海洋大学信息学院）；College of Computer, Beijing University of Posts and Telecommunications（北京邮电大学计算机学院）
Masked Representation Modeling for Domain-Adaptive Segmentation
South China University of Technology（华南理工大学）；School of Electronic and Information Engineering, Wuyi University（武夷大学电子与信息工程学院）；South China Agricultural University（华南农业大学）
3D Cloud reconstruction through geospatially-aware Masked Autoencoders
National Observatory of Athens（雅典国家天文台）；National Technical University of Athens（雅典国立技术大学）；Universitat de València（瓦伦西亚大学）；University of Oxford（牛津大学）；Royal Belgian Institute of Natural Sciences（比利时皇家自然科学研究所）；Imperial College London（伦敦帝国学院）；University of Sussex（萨塞克斯大学）；European Space Agency（欧洲空间局）；CSIC-UCM-IGEO（西班牙高等科研理事会-马德里康普顿斯大学-地球与环境研究所）
本研究结合 MSG/SEVIRI 图像与 CloudSat/CPR 雷达廓线，采用 MAE 及地理空间感知 SatMAE 进行自监督预训练和微调，实现优于 U-Nets 的 3D 云结构重建。
Towards Efficient General Feature Prediction in Masked Skeleton Modeling
Hefei University of Technology（合肥工业大学）；Jilin University（吉林大学）；Zhejiang Gongshang University（浙江工商大学）；University of Science and Technology of China（中国科学技术大学）
Wavelet-Domain Masked Image Modeling for Color-Consistent HDR Video Reconstruction
School of Computer Science and Technology and the Key Laboratory of Embedded System and Service Computing, Ministry of Education, Tongji University（计算机科学与技术学院和嵌入式系统与服务计算重点实验室，教育部，同济大学）；College of Electronic and Information Engineering, the School of Computer Science and Technology, and the Key Laboratory of Embedded System and Service Computing, Ministry of Education, Tongji University（电子与信息工程学院、计算机科学与技术学院和嵌入式系统与服务计算重点实验室，教育部，同济大学）；Pengcheng Laboratory（鹏城实验室）
University of Chinese Academy of Sciences（中国科学院大学）；Meitu Inc.（美图公司）
Contrastive learning (CL) has shown great power in self-supervised learning due to its ability to capture insight correlations among large-scale data. Current CL models are biased to learn only the ability to discriminate positive and negative pairs due to the discriminative task setting. However, this bias would lead to ignoring its sufficiency for other downstream tasks, which we call the discriminative information overfitting problem. In this paper, we propose to tackle the above problems from the aspect of the Information Bottleneck (IB) principle, further pushing forward the frontier of C…
Semantics-Guided Multimodal Masked Autoencoder Pretraining for 3D BEV Object Detection
University of Lincoln, Lincoln Centre for Autonomous Systems（林肯大学，林肯自主系统中心）；University of Cambridge, Institute for Manufacturing, Department of Engineering（剑桥大学，制造研究所，工程系）
Meta Reality Labs（元宇宙现实实验室）；Technical University of Munich（慕尼黑工业大学）
Current popular backbones in computer vision, such as Vision Transformers (ViT) and ResNets are trained to perceive the world from 2D images. However, to more effectively understand 3D structural priors in 2D backbones, we propose Mask3D to leverage existing large-scale RGB-D data in a self-supervised pre-training to embed these 3D priors into 2D learned feature representations. In contrast to traditional 3D contrastive learning paradigms requiring 3D reconstructions or multi-view correspondences, our approach is simple: we formulate a pre-text reconstruction task by masking RGB and depth patc…
Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）
Masked Next-Scale Prediction for Self-supervised Scene Text Recognition
Nankai University（南开大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Department of Automation and BNRist, Tsinghua University（清华大学自动化系和清华大学脑科学与智能技术研究院）
