This paper introduces SenPa-MAE, a transformer architecture that encodes the sensor parameters of an observed multispectral signal into the image embeddings. SenPa-MAE can be pre-trained on imagery of different satellites with non-matching spectral or geometrical sensor characteristics. To incorporate sensor parameters, we propose a versatile sensor parameter encoding module as well as a data augmentation strategy for the diversification of the pre-training dataset. This enables the model to effectively differentiate between various sensors and gain an understanding of sensor parameters and the correlation to the observed signal. Given the rising number of Earth observation satellite missions and the diversity in their sensor specifications, our approach paves the way towards a sensor-independent Earth observation foundation model. This opens up possibilities such as cross-sensor training and sensor-independent inference.
$\mathsf{CSMAE~}$:~Cataract Surgical Masked Autoencoder (MAE) based Pre-training
Johns Hopkins University（约翰斯·霍普金斯大学）；Wilmer Eye Institute, Johns Hopkins University（约翰斯·霍普金斯大学威尔默眼科研究所）；Malone Center for Engineering in Healthcare, Johns Hopkins University（约翰斯·霍普金斯大学马龙医疗保健工程中心）
Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation
National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；School of Computer Science（计算机学院）；Peking University（北京大学）；CUHK（香港中文大学）；University of Science Technology of China（中国科学技术大学）
Advancing Image-Based Grapevine Variety Classification with a New Benchmark and Evaluation of Masked Autoencoders
Engineering Department University of Trás-os-Montes and Alto Douro（特拉斯-奥斯蒙特斯和阿尔托多罗大学工程系）；Fundação para a Salvaguarda e Valorização do Vale do Côa（科阿谷保护与价值基金会）；Universitat de Barcelona（巴塞罗那大学）
USF-MAE: Ultrasound Self-Supervised Foundation Model with Masked Autoencoding
Department of Systems and Computer Engineering, Carleton University（系统与计算机工程系，卡尔顿大学）；Department of Clinical Science and Translational Medicine, University of Ottawa（临床科学与转化医学系，渥太华大学）；Department of Obstetrics and Gynecology, University of Ottawa（妇产科系，渥太华大学）；School of Epidemiology and Public Health, University of Ottawa（流行病学与公共卫生学院，渥太华大学）；Department of Methodological and Implementation Research, Ottawa Hospital Research Institute（方法学与实施研究系，渥太华医院研究学院）；Department of Acute Care Research, Ottawa Hospital Research Institute（急症护理研究系，渥太华医院研究学院）
Contrastive Learning and Masked Image Modelling have demonstrated exceptional performance on self-supervised representation learning, where Momentum Contrast (i.e., MoCo) and Masked AutoEncoder (i.e., MAE) are the state-of-the-art, respectively. In this work, we propose MOMA to distill from pre-trained MoCo and MAE in a self-supervised manner to collaborate the knowledge from both paradigms. We introduce three different mechanisms of knowledge transfer in the propsoed MOMA framework. : (1) Distill pre-trained MoCo to MAE. (2) Distill pre-trained MAE to MoCo (3) Distill pre-trained MoCo and MAE…
Vision Transformer (ViT) suffers from data scarcity in semi-supervised learning (SSL). To alleviate this issue, inspired by masked autoencoder (MAE), which is a data-efficient self-supervised learner, we propose Semi-MAE, a pure ViT-based SSL framework consisting of a parallel MAE branch to assist the visual representation learning and make the pseudo labels more accurate. The MAE branch is designed as an asymmetric architecture consisting of a lightweight decoder and a shared-weights encoder. We feed the weakly-augmented unlabeled data with a high masking ratio to the MAE branch and reconstru…
Trajectory prediction has been a crucial task in building a reliable autonomous driving system by anticipating possible dangers. One key issue is to generate consistent trajectory predictions without colliding. To overcome the challenge, we propose an efficient masked autoencoder for trajectory prediction (Traj-MAE) that better represents the complicated behaviors of agents in the driving environment. Specifically, our Traj-MAE employs diverse masking strategies to pre-train the trajectory encoder and map encoder, allowing for the capture of social and temporal information among agents while l…
Masked Autoencoders (MAE) have been popular paradigms for large-scale vision representation pre-training. However, MAE solely reconstructs the low-level RGB signals after the decoder and lacks supervision upon high-level semantics for the encoder, thus suffering from sub-optimal learned representations and long pre-training epochs. To alleviate this, previous methods simply replace the pixel reconstruction targets of 75% masked tokens by encoded features from pre-trained image-image (DINO) or image-language (CLIP) contrastive learning. Different from those efforts, we propose to Mimic before R…
Masked Autoencoding (MAE) has emerged as an effective approach for pre-training representations across multiple domains. In contrast to discrete tokens in natural languages, the input for image MAE is continuous and subject to additional specifications. We systematically study each input specification during the pre-training stage, and find sequence length is a key axis that further scales MAE. Our study leads to a long-sequence version of MAE with minimal changes to the original recipe, by just decoupling the mask size from the patch size. For object detection and semantic segmentation, our l…
Electronics and Telecommunications Research Institute (ETRI)（电子与电信研究所）；Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；AITRICS
Masked image modeling (MIM) has become a popular strategy for self-supervised learning~(SSL) of visual representations with Vision Transformers. A representative MIM model, the masked auto-encoder (MAE), randomly masks a subset of image patches and reconstructs the masked patches given the unmasked patches. Concurrently, many recent works in self-supervised learning utilize the student/teacher paradigm which provides the student with an additional target based on the output of a teacher composed of an exponential moving average (EMA) of previous students. Although common, relatively little is …
DAP-MAE: Domain-Adaptive Point Cloud Masked Autoencoder for Effective Cross-Domain Learning
School of Computer Science & Software Engineering, Shenzhen University（计算机科学与软件工程学院，深圳大学）；School of Artificial Intelligence, Shenzhen University（人工智能学院，深圳大学）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（大数据系统计算技术国家工程实验室，深圳大学）；Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University（广东省智能信息处理重点实验室，深圳大学）
Deblurring Masked Autoencoder is Better Recipe for Ultrasound Image Recognition
West China Hospital, Sichuan University（四川大学华西医院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Sichuan University（四川大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Vision Transformers (ViTs) outperforms convolutional neural networks (CNNs) in several vision tasks with its global modeling capabilities. However, ViT lacks the inductive bias inherent to convolution making it require a large amount of data for training. This results in ViT not performing as well as CNNs on small datasets like medicine and science. We experimentally found that masked autoencoders (MAE) can make the transformer focus more on the image itself, thus alleviating the data-hungry issue of ViT to some extent. Yet the current MAE model is too complex resulting in over-fitting problem…
Zhejiang University（浙江大学）；Shanghai AI Lab（上海人工智能实验室）；Beihang University（北京航空航天大学）；COMAC Beijing Aircraft Technology Research Institute（中国商飞北京民用飞机技术研究中心）
Despite the tremendous progress of Masked Autoencoders (MAE) in developing vision tasks such as image and video, exploring MAE in large-scale 3D point clouds remains challenging due to the inherent irregularity. In contrast to previous 3D MAE frameworks, which either design a complex decoder to infer masked information from maintained regions or adopt sophisticated masking strategies, we instead propose a much simpler paradigm. The core idea is to apply a \textbf{G}enerative \textbf{D}ecoder for MAE (GD-MAE) to automatically merges the surrounding context to restore the masked geometric knowle…
RI-MAE: Rotation-Invariant Masked AutoEncoders for Self-Supervised Point Cloud Representation Learning
South China University of Technology（华南理工大学）；The University of Adelaide（阿德莱德大学）；The University of Western Australia（西澳大学）；The University of Sydney（悉尼大学）
Hi-End-MAE: Hierarchical encoder-driven masked autoencoders are stronger vision learners for medical image segmentation
University of Science and Technology of China (USTC)（中国科学技术大学）；Suzhou Institute for Advanced Research, USTC（中国科学技术大学苏州高等研究院）；Stanford University（斯坦福大学）；State Key Laboratory of Precision and Intelligent Chemistry, USTC（中国科学技术大学精准与智能化学国家重点实验室）；Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS（中国科学院计算技术研究所中国科学院智能信息处理重点实验室）
Masked Autoencoders (MAE) based on a reconstruction task have risen to be a promising paradigm for self-supervised learning (SSL) and achieve state-of-the-art performance across different benchmark datasets. However, despite its impressive empirical success, there is still limited theoretical understanding of it. In this paper, we propose a theoretical understanding of how masking matters for MAE to learn meaningful features. We establish a close connection between MAE and contrastive learning, which shows that MAE implicit aligns the mask-induced positive pairs. Built upon this connection, we…
Efficient Self-supervised Vision Pretraining with Local Masked Reconstruction
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Monash University（莫纳什大学）；Nile University（尼罗河大学）；Nanyang Technological University（南洋理工大学）
The development of deep learning models in medical image analysis is majorly limited by the lack of large-sized and well-annotated datasets. Unsupervised learning does not require labels and is more suitable for solving medical image analysis problems. However, most of the current unsupervised learning methods need to be applied to large datasets. To make unsupervised learning applicable to small datasets, we proposed Swin MAE, which is a masked autoencoder with Swin Transformer as its backbone. Even on a dataset of only a few thousand medical images and without using any pre-trained models, S…
Washington University in St. Louis（圣路易斯华盛顿大学）；University of Oxford（牛津大学）
"Masked Autoencoders (MAE) Are Scalable Vision Learners" revolutionizes the self-supervised learning method in that it not only achieves the state-of-the-art for image pre-training, but is also a milestone that bridges the gap between visual and linguistic masked autoencoding (BERT-style) pre-trainings. However, to our knowledge, to date there are no theoretical perspectives to explain the powerful expressivity of MAE. In this paper, we, for the first time, propose a unified theoretical framework that provides a mathematical understanding for MAE. Specifically, we explain the patch-based atten…
Learning from the Right Patches: A Two-Stage Wavelet-Driven Masked Autoencoder for Histopathology Representation Learning
PLRI Medical Informatics Institute（PLRI医学信息学研究所）；CAIMed Reserch Center（CAIMed研究中心）；Hannover Medical School（汉诺威医学院）；Computer Science Institute（计算机科学研究所）；Leibniz University Hannover（莱比锡大学汉诺威分校）；Sanford Burnham Prebys Medical Discovery Institute（桑福医疗发现研究所）；Rady Children’s Institute for Genomic Medicine（拉迪儿童基因医学研究所）；University of California San Diego（加州大学圣地亚哥分校）
Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；JD Explore Academy（京东探索研究院）；Renmin University of China（中国人民大学）
Recently, significant progress has been made in masked image modeling to catch up to masked language modeling. However, unlike words in NLP, the lack of semantic decomposition of images still makes masked autoencoding (MAE) different between vision and language. In this paper, we explore a potential visual analogue of words, i.e., semantic parts, and we integrate semantic information into the training process of MAE by proposing a Semantic-Guided Masking strategy. Compared to widely adopted random masking, our masking strategy can gradually guide the network to learn various information, i.e.,…
University of Surrey（萨里大学）；ByteDance（字节跳动）；The University of Hong Kong（香港大学）；Harbin Institute of Technology（哈尔滨工业大学）
Masked autoencoders (MAEs) have emerged recently as art self-supervised spatiotemporal representation learners. Inheriting from the image counterparts, however, existing video MAEs still focus largely on static appearance learning whilst are limited in learning dynamic temporal information hence less effective for video downstream tasks. To resolve this drawback, in this work we present a motion-aware variant -- MotionMAE. Apart from learning to reconstruct individual masked patches of video frames, our model is designed to additionally predict the corresponding motion structure information ov…
A$^{2}$-MAE: A spatial-temporal-spectral unified remote sensing pre-training method based on anchor-aware masked autoencoder
National Supercomputing Center in Shenzhen（深圳国家超级计算中心）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；Sun Yat-Sen University（中山大学）；SenseTime Group（商汤集团）
