Generative self-supervised learning on graphs, particularly graph masked autoencoders, has emerged as a popular learning paradigm and demonstrated its efficacy in handling non-Euclidean data. However, several remaining issues limit the capability of existing methods: 1) the disregard of uneven node significance in masking, 2) the underutilization of holistic graph information, 3) the ignorance of semantic knowledge in the representation space due to the exclusive use of reconstruction loss in the output space, and 4) the unstable reconstructions caused by the large volume of masked contents. In light of this, we propose UGMAE, a unified framework for graph masked autoencoders to address these issues from the perspectives of adaptivity, integrity, complementarity, and consistency. Specifically, we first develop an adaptive feature mask generator to account for the unique significance of nodes and sample informative masks (adaptivity). We then design a ranking-based structure reconstruction objective joint with feature reconstruction to capture holistic graph information and emphasize the topological proximity between neighbors (integrity). After that, we present a bootstrapping-based similarity module to encode the high-level semantic knowledge in the representation space, complementary to the low-level reconstruction in the output space (complementarity). Finally, we build a consistency assurance module to provide reconstruction objectives with extra stabilized consistency targets (consistency). Extensive experiments demonstrate that UGMAE outperforms both contrastive and generative state-of-the-art baselines on several tasks across multiple datasets.
Masked autoencoders are scalable vision learners, as the title of MAE \cite{he2022masked}, which suggests that self-supervised learning (SSL) in vision might undertake a similar trajectory as in NLP. Specifically, generative pretext tasks with the masked prediction (e.g., BERT) have become a de facto standard SSL practice in NLP. By contrast, early attempts at generative methods in vision have been buried by their discriminative counterparts (like contrastive learning); however, the success of mask image modeling has revived the masking autoencoder (often termed denoising autoencoder in the pa…
University of Science and Technology of China（中国科学技术大学）；Institute of Advanced Technology, University of Science and Technology of China（中国科学技术大学先进技术研究院）；Jiaxing University（嘉兴学院）
Large-scale self-supervised pre-training Transformer architecture have significantly boosted the performance for various tasks in natural language processing (NLP) and computer vision (CV). However, there is a lack of researches on processing multivariate time-series by pre-trained Transformer, and especially, current study on masking time-series for self-supervised learning is still a gap. Different from language and image processing, the information density of time-series increases the difficulty of research. The challenge goes further with the invalidity of the previous patch embedding and …
Wearable sensors provide abundant physiological time series, yet the principles governing their predictive utility remain unclear. We hypothesize that temporal resolution is a fundamental axis of representation learning, with different clinical and behavioral outcomes relying on structure at distinct scales. To test this resolution hypothesis, we introduce HiMAE (Hierarchical Masked Autoencoder), a self supervised framework that combines masked autoencoding with a hierarchical convolutional encoder decoder. HiMAE produces multi resolution embeddings that enable systematic evaluation of which t…
This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers. The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram. We find it beneficial to incorporate local window attention in the decoder, as audio spectrograms are highly correlated in lo…
Aalborg University（奥尔堡大学）；University Politehnica of Bucharest（布加勒斯特理工大学）；University of Bucharest（布加勒斯特大学）；Milestone Systems（迈尔斯通系统公司）；SecurifAI（SecurifAI公司）
本文提出 CL-MAE，利用可学习掩码模块在 MAE 自监督训练中生成由易到难的重建课程，并在 ImageNet 及五个下游任务上验证其优于 MAE 的表示学习效果。
University of Notre Dame（圣母大学）；Brandeis University（布兰迪斯大学）
Generative self-supervised learning (SSL), especially masked autoencoders, has become one of the most exciting learning paradigms and has shown great potential in handling graph data. However, real-world graphs are always heterogeneous, which poses three critical challenges that existing methods ignore: 1) how to capture complex graph structure? 2) how to incorporate various node attributes? and 3) how to encode different node positions? In light of this, we study the problem of generative SSL on heterogeneous graphs and propose HGMAE, a novel heterogeneous graph masked autoencoder model to ad…
Washington University in St. Louis（圣路易斯华盛顿大学）；University of Oxford（牛津大学）
"Masked Autoencoders (MAE) Are Scalable Vision Learners" revolutionizes the self-supervised learning method in that it not only achieves the state-of-the-art for image pre-training, but is also a milestone that bridges the gap between visual and linguistic masked autoencoding (BERT-style) pre-trainings. However, to our knowledge, to date there are no theoretical perspectives to explain the powerful expressivity of MAE. In this paper, we, for the first time, propose a unified theoretical framework that provides a mathematical understanding for MAE. Specifically, we explain the patch-based atten…
This paper studies a conceptually simple extension of Masked Autoencoders (MAE) to spatiotemporal representation learning from videos. We randomly mask out spacetime patches in videos and learn an autoencoder to reconstruct them in pixels. Interestingly, we show that our MAE method can learn strong representations with almost no inductive bias on spacetime (only except for patch and positional embeddings), and spacetime-agnostic random masking performs the best. We observe that the optimal masking ratio is as high as 90% (vs. 75% on images), supporting the hypothesis that this ratio is related…
Self-supervised learning (SSL) has been extensively explored in recent years. Particularly, generative SSL has seen emerging success in natural language processing and other AI fields, such as the wide adoption of BERT and GPT. Despite this, contrastive learning-which heavily relies on structural data augmentation and complicated training strategies-has been the dominant approach in graph SSL, while the progress of generative SSL on graphs, especially graph autoencoders (GAEs), has thus far not reached the potential as promised in other fields. In this paper, we identify and examine the issues…
University of Trieste（特里埃斯特大学）；Abdus Salam International Centre for Theoretical Physics (ICTP)（阿布杜斯·萨拉姆国际理论物理学中心（ICTP））；Scuola Internazionale Superiore di Studi Avanzati (SISSA)（国际先进研究高等学院（SISSA））；University of Nova Gorica（诺瓦戈里察大学）；INFN – National Institute for Nuclear Physics（意大利国家核物理研究所（INFN））；ICSC - Centro Nazionale di Ricerca in High Performance Computing（高性能计算国家研究中心（ICSC））；Imperial College London（伦敦帝国理工学院）
Masked Autoencoders (MAE) based on a reconstruction task have risen to be a promising paradigm for self-supervised learning (SSL) and achieve state-of-the-art performance across different benchmark datasets. However, despite its impressive empirical success, there is still limited theoretical understanding of it. In this paper, we propose a theoretical understanding of how masking matters for MAE to learn meaningful features. We establish a close connection between MAE and contrastive learning, which shows that MAE implicit aligns the mask-induced positive pairs. Built upon this connection, we…
Texas A&M University（德克萨斯A&M大学）；University of Georgia（佐治亚大学）；The Hong Kong Polytechnic University（香港理工大学）；Samsung Research America（三星美国研究院）；Rice University（莱斯大学）
We introduce a novel masked graph autoencoder (MGAE) framework to perform effective learning on graph structure data. Taking insights from self-supervised learning, we randomly mask a large proportion of edges and try to reconstruct these missing edges during training. MGAE has two core designs. First, we find that masking a high ratio of the input graph structure, e.g., $70\%$, yields a nontrivial and meaningful self-supervisory task that benefits downstream applications. Second, we employ a graph neural network (GNN) as an encoder to perform message propagation on the partially-masked graph.…
University of Maryland（马里兰大学）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Google Research（谷歌研究院）
We introduce CAN, a simple, efficient and scalable method for self-supervised learning of visual representations. Our framework is a minimal and conceptually clean synthesis of (C) contrastive learning, (A) masked autoencoders, and (N) the noise prediction approach used in diffusion models. The learning mechanisms are complementary to one another: contrastive learning shapes the embedding space across a batch of image samples; masked autoencoders focus on reconstruction of the low-frequency spatial correlations in a single image sample; and noise prediction encourages the reconstruction of the…
Self-supervised learning (SSL) enables learning useful inductive biases through utilizing pretext tasks that require no labels. The unlabeled nature of SSL makes it especially important for whole slide histopathological images (WSIs), where patch-level human annotation is difficult. Masked Autoencoders (MAE) is a recent SSL method suitable for digital pathology as it does not require negative sampling and requires little to no data augmentations. However, the domain shift between natural images and digital pathology images requires further research in designing MAE for patch-level WSIs. In thi…
This paper presents a novel framework for unsupervised anomaly detection on masked objects called ODDObjects, which stands for Out-of-Distribution Detection on Objects. ODDObjects is designed to detect anomalies of various categories using unsupervised autoencoders trained on COCO-style datasets. The method utilizes autoencoder-based image reconstruction, where high reconstruction error indicates the possibility of an anomaly. The framework extends previous work on anomaly detection with autoencoders, comparing state-of-the-art models trained on object recognition datasets. Various model archi…
Pre-Training Graph Contrastive Masked Autoencoders are Strong Distillers for EEG
Department of Electrical and Computer Engineering, Lehigh University, Bethlehem, PA, USA（电气与计算机工程系，莱维大学）；Department of Bioengineering, Lehigh University, Bethlehem, PA, USA（生物工程系，莱维大学）；Center for Neuroscience Research, Children’s National Hospital, Washington, DC, USA（神经科学研究中心，儿童国家医院）；Department of Computer Science and Engineering, Lehigh University, Bethlehem, PA, USA（计算机科学与工程系，莱维大学）
Test-time training adapts to a new test distribution on the fly by optimizing a model for each test input using self-supervision. In this paper, we use masked autoencoders for this one-sample learning problem. Empirically, our simple method improves generalization on many visual benchmarks for distribution shifts. Theoretically, we characterize this improvement in terms of the bias-variance trade-off.
Beyond Labels: A Self-Supervised Framework with Masked Autoencoders and Random Cropping for Breast Cancer Subtype Classification
DISIT, Computer Science Institute, University of Piemonte Orientale（DISIT计算机科学研究所，皮埃蒙特东方大学）；Center for Translational Research on Autoimmune and Allergic Diseases (CAAD), Department of Health Sciences, University of Piemonte Orientale（自身免疫和过敏疾病转化研究中心（CAAD），健康科学系，皮埃蒙特东方大学）；Interdisciplinary Research Center of Autoimmune Diseases (IRCAD), Department of Health Sciences, University of Piemonte Orientale（自身免疫疾病跨学科研究中心（IRCAD），健康科学系，皮埃蒙特东方大学）
提出一种基于 MAE 与随机裁剪的自监督框架，从 WSI 学习组织病理图像表示，并通过线性探针完成乳腺癌多亚型分类。
The Hong Kong University of Science and Technology（香港科技大学）；Beijing Normal University（北京师范大学）
This paper shows that masked autoencoder with extrapolator (ExtraMAE) is a scalable self-supervised model for time series generation. ExtraMAE randomly masks some patches of the original time series and learns temporal dynamics by recovering the masked patches. Our approach has two core designs. First, ExtraMAE is self-supervised. Supervision allows ExtraMAE to effectively and efficiently capture the temporal dynamics of the original time series. Second, ExtraMAE proposes an extrapolator to disentangle two jobs of the decoder: recovering latent representations and mapping them back into the fe…
Masked Graph Autoencoders with Contrastive Augmentation for Spatially Resolved Transcriptomics Data
School of Information Science and Engineering, Yunnan University（云南大学信息科学与工程学院）；College of Nursing Health Sciences, Yunnan Open University（云南开放大学护理与健康科学学院）
Semi-Supervised Masked Autoencoders: Unlocking Vision Transformer Potential with Limited Data
Department of Electrical and Computer Engineering, Rowan University, Glassboro, NJ, USA（电子与计算机工程系，罗文大学，Glassboro, NJ, USA）；Department of Computer Science, Stevens Institute of Technology, Hoboken, NJ, USA（计算机科学系，史蒂文斯理工学院，Hoboken, NJ, USA）
Multi-scale Masked Autoencoder for Electrocardiogram Anomaly Detection
Fuwai Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院阜外医院）；National Center for Cardiovascular Diseases（国家心血管病中心）；National Clinical Research Center for Cardiovascular Diseases（国家心血管疾病临床医学研究中心）
