Due to the prevalence of scale variance in nature images, we propose to use image scale as a self-supervised signal for Masked Image Modeling (MIM). Our method involves selecting random patches from the input image and downsampling them to a low-resolution format. Our framework utilizes the latest advances in super-resolution (SR) to design the prediction head, which reconstructs the input from low-resolution clues and other patches. After 400 epochs of pre-training, our Super Resolution Masked Autoencoders (SRMAE) get an accuracy of 82.1% on the ImageNet-1K task. Image scale signal also allows our SRMAE to capture scale invariance representation. For the very low resolution (VLR) recognition task, our model achieves the best performance, surpassing DeriveNet by 1.3%. Our method also achieves an accuracy of 74.84% on the task of recognizing low-resolution facial expressions, surpassing the current state-of-the-art FMD by 9.48%.
Fraunhofer Institute for Computer Graphics Research IGD（弗劳恩霍夫计算机图形研究所）；TU Darmstadt（达姆施塔特工业大学）
The emergence of the global COVID-19 pandemic poses new challenges for biometrics. Not only are contactless biometric identification options becoming more important, but face recognition has also recently been confronted with the frequent wearing of masks. These masks affect the performance of previous face recognition systems, as they hide important identity information. In this paper, we propose a mask-invariant face recognition solution (MaskInv) that utilizes template-level knowledge distillation within a training paradigm that aims at producing embeddings of masked faces that are similar …
RI-MAE: Rotation-Invariant Masked AutoEncoders for Self-Supervised Point Cloud Representation Learning
South China University of Technology（华南理工大学）；The University of Adelaide（阿德莱德大学）；The University of Western Australia（西澳大学）；The University of Sydney（悉尼大学）
Recently, Masked Image Modeling (MIM) achieves great success in self-supervised visual recognition. However, as a reconstruction-based framework, it is still an open question to understand how MIM works, since MIM appears very different from previous well-studied siamese approaches such as contrastive learning. In this paper, we propose a new viewpoint: MIM implicitly learns occlusion-invariant features, which is analogous to other siamese methods while the latter learns other invariance. By relaxing MIM formulation into an equivalent siamese form, MIM methods can be interpreted in a unified f…
WuHan University（武汉大学）；Zhejiang University（浙江大学）；The University of Sydney（悉尼大学）；SenseTime Research（商汤科技研究院）；Qing Yuan Research Institute（清源研究院）；Shanghai Jiao Tong University（上海交通大学）
Generalizing learned representations across significantly different visual domains is a fundamental yet crucial ability of the human visual system. While recent self-supervised learning methods have achieved good performances with evaluation set on the same domain as the training set, they will have an undesirable performance decrease when tested on a different domain. Therefore, the self-supervised learning from multiple domains task is proposed to learn domain-invariant features that are not only suitable for evaluation on the same domain as the training set but also can be generalized to un…
Shanghai University（上海大学）；JD AI Research（京东人工智能研究院）
Near-infrared to visible (NIR-VIS) face recognition is the most common case in heterogeneous face recognition, which aims to match a pair of face images captured from two different modalities. Existing deep learning based methods have made remarkable progress in NIR-VIS face recognition, while it encounters certain newly-emerged difficulties during the pandemic of COVID-19, since people are supposed to wear facial masks to cut off the spread of the virus. We define this task as NIR-VIS masked face recognition, and find it problematic with the masked face in the NIR probe image. First, the lack…
LADMIM: Logical Anomaly Detection with Masked Image Modeling in Discrete Latent Space
Department of Engineering University of Fukui（工程系冈山大学）；Department of Electronics and Information Engineering National Institute of Technology, Ishikawa College（电子信息工程系石川县国立技术学院）
Huazhong University of Science and Technology（华中科技大学）；La Trobe University（拉筹伯大学）
Transformer framework has been showing superior performances in visual object tracking for its great strength in information aggregation across the template and search image with the well-known attention mechanism. Most recent advances focus on exploring attention mechanism variants for better information aggregation. We find these schemes are equivalent to or even just a subset of the basic self-attention mechanism. In this paper, we prove that the vanilla self-attention structure is sufficient for information aggregation, and structural adaption is unnecessary. The key is not the attention s…
Xiamen University（厦门大学）；The University of Hong Kong（香港大学）
Modern studies in radiograph representation learning rely on either self-supervision to encode invariant semantics or associated radiology reports to incorporate medical expertise, while the complementarity between them is barely noticed. To explore this, we formulate the self- and report-completion as two complementary objectives and present a unified framework based on masked record modeling (MRM). In practice, MRM reconstructs masked image patches and masked report tokens following a multi-task scheme to learn knowledge-enhanced semantic representations. With MRM pre-training, we obtain pre…
CrossVideoMAE: Self-Supervised Image-Video Representation Learning with Masked Autoencoders
University of Moratuwa（莫拉图瓦大学）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Auckland（奥克兰大学）；Weizmann Institute of Science（魏茨曼科学研究所）
HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception
Zhejiang University（浙江大学）；Baidu VIS（百度视觉技术部）；University of Science and Technology Beijing（北京科技大学）；South China University of Technology（华南理工大学）；Shanghai Jiao Tong University（上海交通大学）；Jilin University（吉林大学）
Dual form Complementary Masking for Domain-Adaptive Image Segmentation
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Data Science Institute, Imperial College London（伦敦帝国理工学院数据科学研究院）；iFLYTEK CO., LTD（iFLYTEK公司）
Better with Less: Tackling Heterogeneous Multi-Modal Image Joint Pretraining via Conditioned and Degraded Masked Autoencoder
College of Electronic Science and Technology, National University of Defense Technology (NUDT)（电子科学与技术学院，国防科技大学）；Beijing Institute of Remote Sensing Information（遥感信息研究所）
Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approach
Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Electronic Information School, Wuhan University（武汉大学电子信息学院）
University of California, San Diego（加州大学圣迭戈分校）；University of Maryland（马里兰大学）
We present a self-supervised pre-training approach for learning rich visual language representations for both handwritten and printed historical document transcription. After supervised fine-tuning of our pre-trained encoder representations for low-resource document transcription on two languages, (1) a heterogeneous set of handwritten Islamicate manuscript images and (2) early modern English printed documents, we show a meaningful improvement in recognition accuracy over the same supervised model trained from scratch with as few as 30 line image transcriptions for training. Our masked languag…
Time2General: Learning Spatiotemporal Invariant Representations for Domain-Generalization Video Semantic Segmentation
School of Computer Engineering, Jimei University, Xiamen, China（厦门大学计算机工程学院）；College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China（南京航空航天大学人工智能学院）
University of Amsterdam（阿姆斯特丹大学）；DUniversum（3D宇宙公司）
We investigate the use of photometric invariance and deep learning to compute intrinsic images (albedo and shading). We propose albedo and shading gradient descriptors which are derived from physics-based models. Using the descriptors, albedo transitions are masked out and an initial sparse shading map is calculated directly from the corresponding RGB image gradients in a learning-free unsupervised manner. Then, an optimization method is proposed to reconstruct the full dense shading map. Finally, we integrate the generated shading map into a novel deep learning framework to refine it and also…
Learning and Leveraging World Models in Visual Representation Learning
FAIR at Meta（Meta FAIR实验室）；Univ Gustave Eiffel（Gustave Eiffel大学）；CNRS（法国国家科学研究中心）；INRIA（法国国家信息与自动化研究所）；Courant Institute, New York University（纽约大学柯朗数学科学研究所）；Center for Data Science, New York University（纽约大学数据科学中心）
本文提出 Image World Models，将 JEPA 从掩码预测扩展到潜空间中的全局光度变换预测，并证明其可微调出强性能且能控制表征的不变性或等变性。
City University of Hong Kong（香港城市大学）；Microsoft Cloud + AI（微软云与人工智能部门）
Image completion has made tremendous progress with convolutional neural networks (CNNs), because of their powerful texture modeling capacity. However, due to some inherent properties (e.g., local inductive prior, spatial-invariant kernels), CNNs do not perform well in understanding global structures or naturally support pluralistic completion. Recently, transformers demonstrate their power in modeling the long-term relationship and generating diverse results, but their computation complexity is quadratic to input length, thus hampering the application in processing high-resolution images. This…
