Masked AutoEncoder (MAE) has recently led the trends of visual self-supervision area by an elegant asymmetric encoder-decoder design, which significantly optimizes both the pre-training efficiency and fine-tuning accuracy. Notably, the success of the asymmetric structure relies on the "global" property of Vanilla Vision Transformer (ViT), whose self-attention mechanism reasons over arbitrary subset of discrete image patches. However, it is still unclear how the advanced Pyramid-based ViTs (e.g., PVT, Swin) can be adopted in MAE pre-training as they commonly introduce operators within "local" windows, making it difficult to handle the random sequence of partial vision tokens. In this paper, we propose Uniform Masking (UM), successfully enabling MAE pre-training for Pyramid-based ViTs with locality (termed "UM-MAE" for short). Specifically, UM includes a Uniform Sampling (US) that strictly samples $1$ random patch from each $2 \times 2$ grid, and a Secondary Masking (SM) which randomly masks a portion of (usually $25\%$) the already sampled regions as learnable tokens. US preserves equivalent elements across multiple non-overlapped local windows, resulting in the smooth support for popular Pyramid-based ViTs; whilst SM is designed for better transferable visual representations since US reduces the difficulty of pixel recovery pre-task that hinders the semantic learning. We demonstrate that UM-MAE significantly improves the pre-training efficiency (e.g., it speeds up and reduces the GPU memory by $\sim 2\times$) of Pyramid-based ViTs, but maintains the competitive fine-tuning performance across downstream tasks. For example using HTC++ detector, the pre-trained Swin-Large backbone self-supervised under UM-MAE only in ImageNet-1K can even outperform the one supervised in ImageNet-22K. The codes are available at https://github.com/implus/UM-MAE.
Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；JD Explore Academy（京东探索研究院）；Renmin University of China（中国人民大学）
Recently, significant progress has been made in masked image modeling to catch up to masked language modeling. However, unlike words in NLP, the lack of semantic decomposition of images still makes masked autoencoding (MAE) different between vision and language. In this paper, we explore a potential visual analogue of words, i.e., semantic parts, and we integrate semantic information into the training process of MAE by proposing a Semantic-Guided Masking strategy. Compared to widely adopted random masking, our masking strategy can gradually guide the network to learn various information, i.e.,…
Trajectory prediction has been a crucial task in building a reliable autonomous driving system by anticipating possible dangers. One key issue is to generate consistent trajectory predictions without colliding. To overcome the challenge, we propose an efficient masked autoencoder for trajectory prediction (Traj-MAE) that better represents the complicated behaviors of agents in the driving environment. Specifically, our Traj-MAE employs diverse masking strategies to pre-train the trajectory encoder and map encoder, allowing for the capture of social and temporal information among agents while l…
Zhejiang University（浙江大学）；Shanghai AI Lab（上海人工智能实验室）；Beihang University（北京航空航天大学）；COMAC Beijing Aircraft Technology Research Institute（中国商飞北京民用飞机技术研究中心）
Despite the tremendous progress of Masked Autoencoders (MAE) in developing vision tasks such as image and video, exploring MAE in large-scale 3D point clouds remains challenging due to the inherent irregularity. In contrast to previous 3D MAE frameworks, which either design a complex decoder to infer masked information from maintained regions or adopt sophisticated masking strategies, we instead propose a much simpler paradigm. The core idea is to apply a \textbf{G}enerative \textbf{D}ecoder for MAE (GD-MAE) to automatically merges the surrounding context to restore the masked geometric knowle…
Vision Transformer (ViT) suffers from data scarcity in semi-supervised learning (SSL). To alleviate this issue, inspired by masked autoencoder (MAE), which is a data-efficient self-supervised learner, we propose Semi-MAE, a pure ViT-based SSL framework consisting of a parallel MAE branch to assist the visual representation learning and make the pseudo labels more accurate. The MAE branch is designed as an asymmetric architecture consisting of a lightweight decoder and a shared-weights encoder. We feed the weakly-augmented unlabeled data with a high masking ratio to the MAE branch and reconstru…
Aalborg University（奥尔堡大学）；University Politehnica of Bucharest（布加勒斯特理工大学）；University of Bucharest（布加勒斯特大学）；Milestone Systems（迈尔斯通系统公司）；SecurifAI（SecurifAI公司）
本文提出 CL-MAE，利用可学习掩码模块在 MAE 自监督训练中生成由易到难的重建课程，并在 ImageNet 及五个下游任务上验证其优于 MAE 的表示学习效果。
This paper studies a conceptually simple extension of Masked Autoencoders (MAE) to spatiotemporal representation learning from videos. We randomly mask out spacetime patches in videos and learn an autoencoder to reconstruct them in pixels. Interestingly, we show that our MAE method can learn strong representations with almost no inductive bias on spacetime (only except for patch and positional embeddings), and spacetime-agnostic random masking performs the best. We observe that the optimal masking ratio is as high as 90% (vs. 75% on images), supporting the hypothesis that this ratio is related…
A$^{2}$-MAE: A spatial-temporal-spectral unified remote sensing pre-training method based on anchor-aware masked autoencoder
National Supercomputing Center in Shenzhen（深圳国家超级计算中心）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；Sun Yat-Sen University（中山大学）；SenseTime Group（商汤集团）
SIGMAE: A Spectral-Index-Guided Foundation Model for Multispectral Remote Sensing
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；School of Artificial Intelligence and Automation, Wuhan University of Science and Technology（武汉科技大学人工智能与自动化学院）；School of Electronic Information, Wuhan University of Science and Technology（武汉科技大学电子信息学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）
Structure-Guided Masked Autoencoders for Ultra-High Resolution Scientific Image Understanding
Hokkaido University（北海道大学）；RIKEN Center for Computational Science (R-CCS)（理化学研究所计算科学研究中心（R-CCS））；Oak Ridge National Laboratory（橡树岭国家实验室）；A*STAR（新加坡科技研究局）；Institute of Science Tokyo（东京科学大学）；Japan Synchrotron Radiation Research Institute (JASRI)（日本同步辐射研究所（JASRI））；RIKEN SPring-8 Center（理化学研究所SPring-8中心）
MaskHOI: Robust 3D Hand-Object Interaction Estimation via Masked Pre-training
PCA Lab, School of Intelligence Science and Technology, Nanjing University（PCA实验室，智能科学与技术学院，南京大学）；Nanyang Technological University（南洋理工大学）；Nanjing University of Science and Technology（南京理工大学）；Nankai University（南开大学）
L-MAE: Longitudinal masked auto-encoder with time and severity-aware encoding for diabetic retinopathy progression prediction
LaTIM UMR 1101, Inserm（法国国家健康与医学研究院UMR 1101拉蒂姆研究所）；University of Western Brittany（西布列塔尼大学）；IMT Atlantique（大西洋电信学院）；Lariboisière Hospital, AP-HP（巴黎公立医院集团拉里布瓦西耶尔医院）；University Hospital of Brest（布雷斯特大学医院）；INSERM U1227 Lymphocytes B et Autoimmunite (LBAI)（法国国家健康与医学研究院UMR 1227 B淋巴细胞与自身免疫研究所）
$\mathsf{CSMAE~}$:~Cataract Surgical Masked Autoencoder (MAE) based Pre-training
Johns Hopkins University（约翰斯·霍普金斯大学）；Wilmer Eye Institute, Johns Hopkins University（约翰斯·霍普金斯大学威尔默眼科研究所）；Malone Center for Engineering in Healthcare, Johns Hopkins University（约翰斯·霍普金斯大学马龙医疗保健工程中心）
Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）
Vision Transformers (ViT) become widely-adopted architectures for various vision tasks. Masked auto-encoding for feature pretraining and multi-scale hybrid convolution-transformer architectures can further unleash the potentials of ViT, leading to state-of-the-art performances on image classification, detection and semantic segmentation. In this paper, our ConvMAE framework demonstrates that multi-scale hybrid convolution-transformer can learn more discriminative representations via the mask auto-encoding scheme. However, directly using the original masking strategy leads to the heavy computat…
University of Oxford（牛津大学）；The University of Edinburgh（爱丁堡大学）；The Alan Turing Institute（艾伦·图灵研究所）；DeepMind（深度思维）
We propose ADIOS, a masked image model (MIM) framework for self-supervised learning, which simultaneously learns a masking function and an image encoder using an adversarial objective. The image encoder is trained to minimise the distance between representations of the original and that of a masked image. The masking function, conversely, aims at maximising this distance. ADIOS consistently improves on state-of-the-art self-supervised learning (SSL) methods on a variety of tasks and datasets -- including classification on ImageNet100 and STL10, transfer learning on CIFAR10/100, Flowers102 and …
MedFLIP: Medical Vision-and-Language Self-supervised Fast Pre-Training with Masked Autoencoder
University of Copenhagen（哥本哈根大学）；Tsinghua University（清华大学）；Shanghai Medical Image Insights（上海医学影像洞察）；Harvard University（哈佛大学）；University of Waterloo（滑铁卢大学）
With the development of generative-based self-supervised learning (SSL) approaches like BeiT and MAE, how to learn good representations by masking random patches of the input image and reconstructing the missing information has grown in concern. However, BeiT and PeCo need a "pre-pretraining" stage to produce discrete codebooks for masked patches representing. MAE does not require a pre-training codebook process, but setting pixels as reconstruction targets may introduce an optimization gap between pre-training and downstream tasks that good reconstruction quality may not always lead to the hi…
Shanghai AI Laboratory（上海人工智能实验室）；Peking University（北京大学）
Pre-training by numerous image data has become de-facto for robust 2D representations. In contrast, due to the expensive data acquisition and annotation, a paucity of large-scale 3D datasets severely hinders the learning for high-quality 3D features. In this paper, we propose an alternative to obtain superior 3D representations from 2D pre-trained models via Image-to-Point Masked Autoencoders, named as I2P-MAE. By self-supervised pre-training, we leverage the well learned 2D knowledge to guide 3D masked autoencoding, which reconstructs the masked point tokens with an encoder-decoder architectu…
Non-Contact Physiological Monitoring in Pediatric Intensive Care Units via Adaptive Masking and Self-Supervised Learning
Biomedical Information Processing Laboratory, École de Technologie Supérieure, University of Quebec（生物医学信息处理实验室，高等技术学院，魁北克大学）；Pediatric Intensive Care Unit, CHU Sainte-Justine（儿童重症监护单元，圣朱斯特医院）
AFFMAE: Scalable Vision Pre-Training for High-Resolution Microscopy Segmentation on Desktop Hardware
Department of Laboratory Medicine & Pathology, University of Washington（华盛顿大学检验医学与病理学系）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Electrical Engineering & Computer Science, Oregon State University（俄勒冈州立大学电气工程与计算机科学系）
This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yield…
Point-SRA: Self-Representation Alignment for 3D Representation Learning
School of Electronics and Information, Xi’an Polytechnic University（西安理工大学电子与信息学院）；School of Software, Xi’an Jiaotong University（西安交通大学软件学院）；School of Computer Science, Xi’an Polytechnic University（西安理工大学计算机科学学院）
Scalable Audio-Visual Masked Autoencoders for Efficient Affective Video Facial Analysis
School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Cyber Science and Engineering, Zhengzhou University（郑州大学网络科学与工程学院）；School of Software, Northwestern Polytechnical University（西北工业大学软件学院）；Institute of Advanced Technology, University of Science and Technology of China（中国科学技术大学先进技术研究院）；Inspur Group（Inspur集团）；Research Center for Space Computing System, Zhejiang Lab（浙江实验室空间计算系统研究中心）
针对情感视频面部分析数据有限及跨模态建模难题，提出可扩展音视频 MAE 模型 AVF-MAE++，通过双重掩码、相关性学习模块和三阶段训练，在17个数据集上取得SOTA。
We explore the plain, non-hierarchical Vision Transformer (ViT) as a backbone network for object detection. This design enables the original ViT architecture to be fine-tuned for object detection without needing to redesign a hierarchical backbone for pre-training. With minimal adaptations for fine-tuning, our plain-backbone detector can achieve competitive results. Surprisingly, we observe: (i) it is sufficient to build a simple feature pyramid from a single-scale feature map (without the common FPN design) and (ii) it is sufficient to use window attention (without shifting) aided with very f…
The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；Centre for Perceptual and Interactive Intelligence Limited（感知与交互智能中心有限公司）
Masked Autoencoders (MAE) have shown great potentials in self-supervised pre-training for language and 2D image transformers. However, it still remains an open question on how to exploit masked autoencoding for learning 3D representations of irregular point clouds. In this paper, we propose Point-M2AE, a strong Multi-scale MAE pre-training framework for hierarchical self-supervised learning of 3D point clouds. Unlike the standard transformer in MAE, we modify the encoder and decoder into pyramid architectures to progressively model spatial geometries and capture both fine-grained and high-leve…
Probabilistic Hyper-Graphs using Multiple Randomly Masked Autoencoders for Semi-supervised Multi-modal Multi-task Learning
Faculty of Automatic Control and Computer Science, Politehnica University of Bucharest（自动化控制与计算机科学系，布加勒斯特理工大学）；Institute of Mathematics of the Romanian Academy（罗马尼亚科学院数学研究所）
