Masked language modeling has become a standard pretraining objective for training encoder-based language models. In this approach, certain tokens in the input are masked, and the model learns to predict them using the surrounding context. This process enables the model to capture both syntactic and semantic properties of language. Conventionally, the tokens selected for masking are chosen at random, which may not always yield the most effective learning signals. In this work, we examine a token masking strategy based on entropy distribution. We use the model's entropy over token predictions to identify which tokens should be masked. This method aims to target tokens that are more informative and uncertain to improve the training efficacy. We also propose a novel self-masking approach that enhances training efficiency without relying on an external reference model. Experimental results demonstrate that our method achieves an average performance improvement of 5% in GLUE scores compared to the baseline. Further, we experiment with combining knowledge distillation with entropy masking, resulting in the best overall results.
MedFLIP: Medical Vision-and-Language Self-supervised Fast Pre-Training with Masked Autoencoder
University of Copenhagen（哥本哈根大学）；Tsinghua University（清华大学）；Shanghai Medical Image Insights（上海医学影像洞察）；Harvard University（哈佛大学）；University of Waterloo（滑铁卢大学）
Institute of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；Microsoft Research Asia（微软亚洲研究院）
Transformer-based autoregressive (AR) methods have achieved appealing performance for varied sequence-to-sequence generation tasks, e.g., neural machine translation, summarization, and code generation, but suffer from low inference efficiency. To speed up the inference stage, many non-autoregressive (NAR) strategies have been proposed in the past few years. Among them, the conditional masked language model (CMLM) is one of the most versatile frameworks, as it can support many different sequence generation scenarios and achieve very competitive performance on these tasks. In this paper, we furt…
Microsoft Dynamics 365 AI（微软Dynamics 365人工智能）；Microsoft Research（微软研究院）
Masked Language Model (MLM) framework has been widely adopted for self-supervised language pre-training. In this paper, we argue that randomly sampled masks in MLM would lead to undesirably large gradient variance. Thus, we theoretically quantify the gradient variance via correlating the gradient covariance with the Hamming distance between two different masks (given a certain text sequence). To reduce the variance due to the sampling of masks, we propose a fully-explored masking strategy, where a text sequence is divided into a certain number of non-overlapping segments. Thereafter, the token…
Masked Autoencoders (MAEs) learn generalizable representations for image, text, audio, video, etc., by reconstructing masked input data from tokens of the visible data. Current MAE approaches for videos rely on random patch, tube, or frame-based masking strategies to select these tokens. This paper proposes AdaMAE, an adaptive masking strategy for MAEs that is end-to-end trainable. Our adaptive masking strategy samples visible tokens based on the semantic context using an auxiliary sampling network. This network estimates a categorical distribution over spacetime-patch tokens. The tokens that …
Resource-Efficient Multiview Perception: Integrating Semantic Masking with Masked Autoencoders
Faculty of Engineering, The University of Sydney, Australia（悉尼大学工程学院）；School of Computer, Data & Mathematical Sciences, Western Sydney University, Australia（西悉尼大学计算机、数据与数学科学学院）
Dual form Complementary Masking for Domain-Adaptive Image Segmentation
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Data Science Institute, Imperial College London（伦敦帝国理工学院数据科学研究院）；iFLYTEK CO., LTD（iFLYTEK公司）
Aalborg University（奥尔堡大学）；University Politehnica of Bucharest（布加勒斯特理工大学）；University of Bucharest（布加勒斯特大学）；Milestone Systems（迈尔斯通系统公司）；SecurifAI（SecurifAI公司）
本文提出 CL-MAE，利用可学习掩码模块在 MAE 自监督训练中生成由易到难的重建课程，并在 ImageNet 及五个下游任务上验证其优于 MAE 的表示学习效果。
Google Brain Robotics（谷歌大脑机器人部门）；Columbia University（哥伦比亚大学）；University of Cambridge（剑桥大学）；University of Oxford（牛津大学）；Google Research（谷歌研究院）；The Alan Turing Institute（艾伦·图灵研究所）；Google（谷歌）
In this paper we provide, to the best of our knowledge, the first comprehensive approach for incorporating various masking mechanisms into Transformers architectures in a scalable way. We show that recent results on linear causal attention (Choromanski et al., 2021) and log-linear RPE-attention (Luo et al., 2021) are special cases of this general mechanism. However by casting the problem as a topological (graph-based) modulation of unmasked attention, we obtain several results unknown before, including efficient d-dimensional RPE-masking and graph-kernel masking. We leverage many mathematical …
Vision-and-Language Pre-training (VLP) improves model performance for downstream tasks that require image and text inputs. Current VLP approaches differ on (i) model architecture (especially image embedders), (ii) loss functions, and (iii) masking policies. Image embedders are either deep models like ResNet or linear projections that directly feed image-pixels into the transformer. Typically, in addition to the Masked Language Modeling (MLM) loss, alignment-based objectives are used for cross-modality interaction, and RoI feature regression and classification tasks for Masked Image-Region Mode…
We study how masking and predicting tokens in an unsupervised fashion can give rise to linguistic structures and downstream performance gains. Recent theories have suggested that pretrained language models acquire useful inductive biases through masks that implicitly act as cloze reductions for downstream tasks. While appealing, we show that the success of the random masking strategy used in practice cannot be explained by such cloze-like masks alone. We construct cloze-like masks using task-specific lexicons for three different classification datasets and show that the majority of pretrained …
Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Alibaba Group（阿里巴巴集团）；Computational Bioscience Research Center, KAUST（阿卜杜拉国王科技大学计算生物科学研究中心）
Masked Latent Transformer with the Random Masking Ratio to Advance the Diagnosis of Dental Fluorosis
State Key Laboratory of Public Big Data, Guizhou University（贵州大学公共大数据国家重点实验室）；College of Computer Science and Technology, Guizhou University（贵州大学计算机科学与技术学院）
This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers. The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram. We find it beneficial to incorporate local window attention in the decoder, as audio spectrograms are highly correlated in lo…
East China Normal University（华东师范大学）；Nanyang Technological University（南洋理工大学）；Alibaba Group（阿里巴巴集团）
Modern face recognition systems (FRS) still fall short when the subjects are wearing facial masks, a common theme in the age of respiratory pandemics. An intuitive partial remedy is to add a mask detector to flag any masked faces so that the FRS can act accordingly for those low-confidence masked faces. In this work, we set out to investigate the potential vulnerability of such FRS equipped with a mask detector, on large-scale masked faces, which might trigger a serious risk, e.g., letting a suspect evade the FRS where both facial identity and mask are undetected. As existing face recognizers …
How to Mask in Error Correction Code Transformer: Systematic and Double Masking
Seoul National University（首尔大学）；University of Ulsan（蔚山大学）；Sungkyunkwan University（成均馆大学）；Pohang University of Science and Technology (POSTECH)（浦项科技大学（POSTECH））
