Cross-Domain Few-Shot Learning (CDFSL) requires the model to transfer knowledge from the data-abundant source domain to data-scarce target domains for fast adaptation, where the large domain gap makes CDFSL a challenging problem. Masked Autoencoder (MAE) excels in effectively using unlabeled data and learning image's global structures, enhancing model generalization and robustness. However, in the CDFSL task with significant domain shifts, we find MAE even shows lower performance than the baseline supervised models. In this paper, we first delve into this phenomenon for an interpretation. We find that MAE tends to focus on low-level domain information during reconstructing pixels while changing the reconstruction target to token features could mitigate this problem. However, not all features are beneficial, as we then find reconstructing high-level features can hardly improve the model's transferability, indicating a trade-off between filtering domain information and preserving the image's global structure. In all, the reconstruction target matters for the CDFSL task. Based on the above findings and interpretations, we further propose Domain-Agnostic Masked Image Modeling (DAMIM) for the CDFSL task. DAMIM includes an Aggregated Feature Reconstruction module to automatically aggregate features for reconstruction, with balanced learning of domain-agnostic information and images' global structure, and a Lightweight Decoder module to further benefit the encoder's generalizability. Experiments on four CDFSL datasets demonstrate that our method achieves state-of-the-art performance.
MM-3DScene: 3D Scene Understanding by Customizing Masked Modeling with Informative-Preserved Reconstruction and Self-Distilled Consistency
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Shenzhen Institutes of Advanced Technology, Chinese Academy of Science（中国科学院深圳先进技术研究院）；China Telecom Corporation Ltd. Data&AI Technology Company（中国电信股份有限公司数据与AI科技公司）；University of Rochester（罗切斯特大学）；University of Science Technology of China（中国科学技术大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Masked Autoencoders (MAE) have been prevailing paradigms for large-scale vision representation pre-training. By reconstructing masked image patches from a small portion of visible image regions, MAE forces the model to infer semantic correlation within an image. Recently, some approaches apply semantic-rich teacher models to extract image features as the reconstruction target, leading to better performance. However, unlike the low-level features such as pixel values, we argue the features extracted by powerful teacher models already encode rich semantic correlation across regions in an intact …
Huazhong University of Science & Technology（华中科技大学）；Tencent PCG（腾讯平台与内容事业群）；Foundation Technology Center（基础技术中心）
Both masked image modeling (MIM) and natural language supervision have facilitated the progress of transferable visual pre-training. In this work, we seek the synergy between two paradigms and study the emerging properties when MIM meets natural language supervision. To this end, we present a novel masked visual Reconstruction In Language semantic Space (RILS) pre-training framework, in which sentence representations, encoded by the text encoder, serve as prototypes to transform the vision-only signals into patch-sentence probabilities as semantically meaningful MIM reconstruction targets. The…
Efficient Self-supervised Vision Pretraining with Local Masked Reconstruction
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Monash University（莫纳什大学）；Nile University（尼罗河大学）；Nanyang Technological University（南洋理工大学）
Shanghai AI Laboratory（上海人工智能实验室）；Simon Fraser University（西蒙弗雷泽大学）；The Chinese University of Hong Kong（香港中文大学）
Masked Image Modeling (MIM) has achieved promising progress with the advent of Masked Autoencoders (MAE) and BEiT. However, subsequent works have complicated the framework with new auxiliary tasks or extra pre-trained models, inevitably increasing computational overhead. This paper undertakes a fundamental analysis of MIM from the perspective of pixel reconstruction, which examines the input image patches and reconstruction target, and highlights two critical but previously overlooked bottlenecks. Based on this analysis, we propose a remarkably simple and effective method, {\ourmethod}, that e…
Masked Image Modeling (MIM) achieves outstanding success in self-supervised representation learning. Unfortunately, MIM models typically have huge computational burden and slow learning process, which is an inevitable obstacle for their industrial applications. Although the lower layers play the key role in MIM, existing MIM models conduct reconstruction task only at the top layer of encoder. The lower layers are not explicitly guided and the interaction among their patches is only used for calculating new activations. Considering the reconstruction task requires non-trivial inter-patch intera…
Exploring Adaptive Masked Reconstruction for Self-Supervised Skeleton-Based Action Recognition
Hefei University of Technology（合肥工业大学）；Jilin University（吉林大学）；Zhejiang Gongshang University（浙江工商大学）；University of Science and Technology of China（中国科学技术大学）
Graduate School of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术研究院）；Faculty of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术学院）
Masked Registration and Autoencoding of CT Images for Predictive Tibia Reconstruction
Technical University Berlin（技术大学柏林）；Charité – Universitätsmedizin Berlin（柏林查理医院）；Freie Universität Berlin（自由大学柏林）；Humboldt-Universität zu Berlin（洪堡大学柏林）；Institute of Medical Informatics（医学信息学研究所）；Center for Musculoskeletal Surgery（骨科手术中心）；Julius Wolff Institute（朱利叶斯·沃尔夫研究所）
Deep learning has thrived by training on large-scale datasets. However, in robotics applications sample efficiency is critical. We propose a novel adaptive masked proxies method that constructs the final segmentation layer weights from few labelled samples. It utilizes multi-resolution average pooling on base embeddings masked with the label to act as a positive proxy for the new class, while fusing it with the previously learned class signatures. Our method is evaluated on PASCAL-$5^i$ dataset and outperforms the state-of-the-art in the few-shot semantic segmentation. Unlike previous methods,…
Unsupervised Anomaly Detection via Masked Diffusion Posterior Sampling
School of Automation Engineering, University of Electronic Science and Technology of China(UESTC)（电子科技大学自动化工程学院）；Shenzhen Institute for Advanced Study, UESTC（电子科技大学深圳高等研究院）
The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；Carnegie Mellon University（卡内基梅隆大学）
This paper introduces the Masked Voxel Jigsaw and Reconstruction (MV-JAR) method for LiDAR-based self-supervised pre-training and a carefully designed data-efficient 3D object detection benchmark on the Waymo dataset. Inspired by the scene-voxel-point hierarchy in downstream 3D object detectors, we design masking and reconstruction strategies accounting for voxel distributions in the scene and local point distributions within the voxel. We employ a Reversed-Furthest-Voxel-Sampling strategy to address the uneven distribution of LiDAR points and propose MV-JAR, which combines two techniques for …
Accelerating Conditional Prompt Learning via Masked Image Modeling for Vision-Language Models
Convergence Research Institute, Sungkyunkwan University（convergence research institute, 首尔大学）；School of Information Technology, Deakin University（信息科技学院, 德金大学）；Department of Electrical and Computer Engineering, Sungkyunkwan University（电气与计算机工程系, 首尔大学）
Unified vision-language frameworks have greatly advanced in recent years, most of which adopt an encoder-decoder architecture to unify image-text tasks as sequence-to-sequence generation. However, existing video-language (VidL) models still require task-specific designs in model architecture and training objectives for each task. In this work, we explore a unified VidL framework LAVENDER, where Masked Language Modeling (MLM) is used as the common interface for all pre-training and downstream tasks. Such unification leads to a simplified model architecture, where only a lightweight MLM head, in…
Unsupervised Anomaly Detection in Medical Images with a Memory-augmented Multi-level Cross-attentional Masked Autoencoder
Harvard University（哈佛大学）；Singapore Management University（新加坡管理大学）；Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）；South Australian Health and Medical Research Institute（南澳大利亚健康与医学研究所）；University of Surrey（萨里大学）
State-of-the-art computer vision models are mostly trained with supervised learning using human-labeled images, which limits their scalability due to the expensive annotation cost. While self-supervised representation learning has achieved impressive progress, it still requires a second stage of finetuning on labeled data. On the other hand, models pre-trained with large-scale text-image supervision (e.g., CLIP) have enabled zero-shot transfer to downstream image classification tasks. However, the zero-shot performance of CLIP-like models are often insufficient for real-world adoption. In this…
Pre-training Point Cloud Compact Model with Partial-aware Reconstruction
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Research Center of Artificial Intelligence, Peng Cheng Laboratory（鹏城实验室人工智能研究中心）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）
Wavelet-Driven Masked Image Modeling: A Path to Efficient Visual Representation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Peng Cheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Jiao Tong University（上海交通大学）
Computed Tomography (CT) is an imaging technique where information about an object are collected at different angles (called projections or scans). Then the cross-sectional image showing the internal structure of the slice is produced by solving an inverse problem. Limited by certain factors such as radiation dosage, projection angles, the produced images can be noisy or contain artifacts. Inspired by the success of transformer for natural language processing, the core idea of this preliminary study is to consider a projection of tomography as a word token, and the whole scan of the cross-sect…
MambaMIR: An Arbitrary-Masked Mamba for Joint Medical Image Reconstruction and Uncertainty Estimation
Imperial College London（伦敦帝国学院）；Royal Brompton Hospital（皇家布朗普顿医院）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；University of Cambridge（剑桥大学）
XJTLU（西交利物浦大学）；University of Liverpool（利物浦大学）；Dinnar Automation Technology（迪纳自动化技术公司）
Few-shot segmentation has been attracting a lot of attention due to its effectiveness to segment unseen object classes with a few annotated samples. Most existing approaches use masked Global Average Pooling (GAP) to encode an annotated support image to a feature vector to facilitate query image segmentation. However, this pipeline unavoidably loses some discriminative information due to the average operation. In this paper, we propose a simple but effective self-guided learning approach, where the lost critical information is mined. Specifically, through making an initial prediction for the a…
SIDME: Self-supervised Image Demoiréing via Masked Encoder-Decoder Reconstruction
College of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；College of Electronic Information and Electrical Engineering, Shanghai Jiaotong University（上海交通大学电子信息与电气工程学院）；College of Information, Shanghai Ocean University（上海海洋大学信息学院）；College of Computer, Beijing University of Posts and Telecommunications（北京邮电大学计算机学院）
MLIP: Medical Language-Image Pre-training with Masked Local Representation Learning
Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院保罗·C·劳特伯生物医学成像研究中心）；Peng Cheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）；The University of Hong Kong（香港大学）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））
