Full-reference image quality metrics (FR-IQMs) aim to measure the visual differences between a pair of reference and distorted images, with the goal of accurately predicting human judgments. However, existing FR-IQMs, including traditional ones like PSNR and SSIM and even perceptual ones such as HDR-VDP, LPIPS, and DISTS, still fall short in capturing the complexities and nuances of human perception. In this work, rather than devising a novel IQM model, we seek to improve upon the perceptual quality of existing FR-IQM methods. We achieve this by considering visual masking, an important characteristic of the human visual system that changes its sensitivity to distortions as a function of local image content. Specifically, for a given FR-IQM metric, we propose to predict a visual masking model that modulates reference and distorted images in a way that penalizes the visual errors based on their visibility. Since the ground truth visual masks are difficult to obtain, we demonstrate how they can be derived in a self-supervised manner solely based on mean opinion scores (MOS) collected from an FR-IQM dataset. Our approach results in enhanced FR-IQM metrics that are more in line with human prediction both visually and quantitatively.
Hebei University of Technology（河北工业大学）；Shenzhen University（深圳大学）；TU Wien（维也纳技术大学）；University of Oxford（牛津大学）
Existing self-supervised learning methods based on contrastive learning and masked image modeling have demonstrated impressive performances. However, current masked image modeling methods are mainly utilized in natural images, and their applications in medical images are relatively lacking. Besides, their fixed high masking strategy limits the upper bound of conditional mutual information, and the gradient noise is considerable, making less the learned representation information. Motivated by these limitations, in this paper, we propose masked patches selection and adaptive masking strategy ba…
Carnegie Mellon University（卡内基梅隆大学）；Amazon Web Services（亚马逊云科技）；Amazon（亚马逊）
Recent methods in self-supervised learning have demonstrated that masking-based pretext tasks extend beyond NLP, serving as useful pretraining objectives in computer vision. However, existing approaches apply random or ad hoc masking strategies that limit the difficulty of the reconstruction task and, consequently, the strength of the learnt representations. We improve upon current state-of-the-art work in learning adversarial masks by proposing a new framework that generates masks in a sequential fashion with different constraints on the adversary. This leads to improvements in performance on…
Self-supervised learning has become a popular approach in recent years for its ability to learn meaningful representations without the need for data annotation. This paper proposes a novel image augmentation technique, overlaying images, which has not been widely applied in self-supervised learning. This method is designed to provide better guidance for the model to understand underlying information, resulting in more useful representations. The proposed method is evaluated using contrastive learning, a widely used self-supervised learning method that has shown solid performance in downstream …
Cooperative Medianet Innovation Center（未来媒体网络协同创新中心）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
Recently, anomaly detection and localization in multimedia data have received significant attention among the machine learning community. In real-world applications such as medical diagnosis and industrial defect detection, anomalies only present in a fraction of the images. To extend the reconstruction-based anomaly detection architecture to the localized anomalies, we propose a self-supervised learning approach through random masking and then restoring, named Self-Supervised Masking (SSM) for unsupervised anomaly detection and localization. SSM not only enhances the training of the inpaintin…
Visual domain adaptation (DA) seeks to transfer trained models to unseen, unlabeled domains across distribution shift, but approaches typically focus on adapting convolutional neural network architectures initialized with supervised ImageNet representations. In this work, we shift focus to adapting modern architectures for object recognition -- the increasingly popular Vision Transformer (ViT) -- and modern pretraining based on self-supervised learning (SSL). Inspired by the design of recent SSL approaches based on learning from partial image inputs generated via masking or cropping -- either …
University of Oxford（牛津大学）；The University of Edinburgh（爱丁堡大学）；The Alan Turing Institute（艾伦·图灵研究所）；DeepMind（深度思维）
We propose ADIOS, a masked image model (MIM) framework for self-supervised learning, which simultaneously learns a masking function and an image encoder using an adversarial objective. The image encoder is trained to minimise the distance between representations of the original and that of a masked image. The masking function, conversely, aims at maximising this distance. ADIOS consistently improves on state-of-the-art self-supervised learning (SSL) methods on a variety of tasks and datasets -- including classification on ImageNet100 and STL10, transfer learning on CIFAR10/100, Flowers102 and …
Max-Planck-Institut für Informatik（马克斯·普朗克计算机科学研究所）；Universidad de Zaragoza（萨拉戈萨大学）
High Dynamic Range (HDR) content is becoming ubiquitous due to the rapid development of capture technologies. Nevertheless, the dynamic range of common display devices is still limited, therefore tone mapping (TM) remains a key challenge for image visualization. Recent work has demonstrated that neural networks can achieve remarkable performance in this task when compared to traditional methods, however, the quality of the results of these learning-based methods is limited by the training data. Most existing works use as training set a curated selection of best-performing results from existing…
Physics-guided deep learning (PG-DL) via algorithm unrolling has received significant interest for improved image reconstruction, including MRI applications. These methods unroll an iterative optimization algorithm into a series of regularizer and data consistency units. The unrolled networks are typically trained end-to-end using a supervised approach. Current supervised PG-DL approaches use all of the available sub-sampled measurements in their data consistency units. Thus, the network learns to fit the rest of the measurements. In this study, we propose to improve the performance and robust…
Collecting large-scale datasets is crucial for training deep models, annotating the data, however, inevitably yields noisy labels, which poses challenges to deep learning algorithms. Previous efforts tend to mitigate this problem via identifying and removing noisy samples or correcting their labels according to the statistical properties (e.g., loss values) among training samples. In this paper, we aim to tackle this problem from a new perspective, delving into the deep feature maps, we empirically find that models trained with clean and mislabeled samples manifest distinguishable activation f…
One of the key issues in facial expression recognition in the wild (FER-W) is that curating large-scale labeled facial images is challenging due to the inherent complexity and ambiguity of facial images. Therefore, in this paper, we propose a self-supervised simple facial landmark encoding (SimFLE) method that can learn effective encoding of facial landmarks, which are important features for improving the performance of FER-W, without expensive labels. Specifically, we introduce novel FaceMAE module for this purpose. FaceMAE reconstructs masked facial images with elaborately designed semantic …
NavInfo Europe（四维图新欧洲公司）；Eindhoven University of Technology（埃因霍温理工大学）
Self-supervised monocular depth estimation is a salient task for 3D scene understanding. Learned jointly with monocular ego-motion estimation, several methods have been proposed to predict accurate pixel-wise depth without using labeled data. Nevertheless, these methods focus on improving performance under ideal conditions without natural or digital corruptions. The general absence of occlusions is assumed even for object-specific depth estimation. These methods are also vulnerable to adversarial attacks, which is a pertinent concern for their reliable deployment in robots and autonomous drivi…
This paper introduces a novel approach to leverage features learned from both supervised and self-supervised paradigms, to improve image classification tasks, specifically for vehicle classification. Two state-of-the-art self-supervised learning methods, DINO and data2vec, were evaluated and compared for their representation learning of vehicle images. The former contrasts local and global views while the latter uses masked prediction on multi-layered representations. In the latter case, supervised learning is employed to finetune a pretrained YOLOR object detector for detecting vehicle wheels…
AIM: Amending Inherent Interpretability via Self-Supervised Masking
Max-Planck-Institute for Informatics（马克斯·普朗克研究所（信息学））；RTG Neuroexplicit Models of Language, Vision, and Action（神经显性语言、视觉和行动模型研究组）；Data and Web Science Group（数据与网络科学小组）；University of Mannheim（曼海姆大学）
Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）
AMLP: Adjustable Masking Lesion Patches for Self-Supervised Medical Image Segmentation
State Key Laboratory of Reliability and Intelligence of Electrical Equipment, School of Health Sciences and Biomedical Engineering, Hebei University of Technology, Tianjin, China（河北工业大学可靠性与智能电气设备国家重点实验室，健康科学与生物医学工程学院，中国天津）；Department of Computer Science, University of Oxford, Oxford, United Kingdom（英国牛津大学计算机科学系）；institute of Logic and Computation, Vienna University of Technology, Vienna, Austria（奥地利技术大学逻辑与计算研究所）
Harnessing Self-Supervised Deep Learning and Geostationary Remote Sensing for Advancing Wildfire and Associated Air Quality Monitoring: Improved Smoke and Fire Front Masking using GOES and TEMPO Radiance Data
Spatial Informatics Group, LLC.（空间信息集团有限责任公司）；ADNET systems, INC.（ADNET系统公司）；NASA Langley Research Center（美国国家航空航天局兰利研究中心）；University at Albany（奥尔巴尼大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；The University of Sydney（悉尼大学）；SenseTime（商汤科技）；Chinese Academy of Sciences - Artificial Intelligence Research（中国科学院人工智能创新研究院）
Semantic segmentation has achieved huge progress via adopting deep Fully Convolutional Networks (FCN). However, the performance of FCN based models severely rely on the amounts of pixel-level annotations which are expensive and time-consuming. To address this problem, it is a good choice to learn to segment with weak supervision from bounding boxes. How to make full use of the class-level and region-level supervisions from bounding boxes is the critical challenge for the weakly supervised learning task. In this paper, we first introduce a box-driven class-wise masking model (BCM) to remove irr…
Non-Contact Physiological Monitoring in Pediatric Intensive Care Units via Adaptive Masking and Self-Supervised Learning
Biomedical Information Processing Laboratory, École de Technologie Supérieure, University of Quebec（生物医学信息处理实验室，高等技术学院，魁北克大学）；Pediatric Intensive Care Unit, CHU Sainte-Justine（儿童重症监护单元，圣朱斯特医院）
Energy-Regularized Spatial Masking: A Novel Approach to Enhancing Robustness and Interpretability in Vision Models
DAVID Lab, UVSQ, Paris-Saclay University（DAVID实验室，UVSQ，巴黎-萨克雷大学）；LIPN, UMR CNRS 7030, Sorbonne Paris Nord University（LIPN，UMR CNRS 7030，索邦巴黎北大学）；LISN, Paris-Saclay University（LISN，巴黎-萨克雷大学）
With the advent of recent advances in unsupervised learning, efficient training of a deep network for image denoising without pairs of noisy and clean images has become feasible. However, most current unsupervised denoising methods are built on the assumption of zero-mean noise under the signal-independent condition. This assumption causes blind denoising techniques to suffer brightness shifting problems on images that are greatly corrupted by extreme noise such as salt-and-pepper noise. Moreover, most blind denoising methods require a random masking scheme for training to ensure the invarianc…
Technical University of Munich（慕尼黑工业大学）；Sharif University of Technology（谢里夫理工大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
The need for a large amount of labeled data in the supervised setting has led recent studies to utilize self-supervised learning to pre-train deep neural networks using unlabeled data. Many self-supervised training strategies have been investigated especially for medical datasets to leverage the information available in the much fewer unlabeled data. One of the fundamental strategies in image-based self-supervision is context prediction. In this approach, a model is trained to reconstruct the contents of an arbitrary missing region of an image based on its surroundings. However, the existing m…
University of Surrey（萨里大学）；iFlyTek-Surrey Joint Research Centre on Artificial Intelligence（讯飞-萨里人工智能联合研究中心）；Surrey Institute for People-Centred Artificial Intelligence（萨里以人为本人工智能研究所）
Existing temporal action detection (TAD) methods rely on a large number of training data with segment-level annotations. Collecting and annotating such a training set is thus highly expensive and unscalable. Semi-supervised TAD (SS-TAD) alleviates this problem by leveraging unlabeled videos freely available at scale. However, SS-TAD is also a much more challenging problem than supervised TAD, and consequently much under-studied. Prior SS-TAD methods directly combine an existing proposal-based TAD method and a SSL method. Due to their sequential localization (e.g, proposal generation) and class…
Joint Low-level and High-level Textual Representation Learning with Multiple Masking Strategies
Wenzhou University Artificial Intelligence and Advanced Manufacturing Institute (AIAMI)（温州大学人工智能与先进制造研究院）；Graduate School of Engineering, Tohoku University（东京东京大学工学研究科）
Single view depth estimation models can be trained from video footage using a self-supervised end-to-end approach with view synthesis as the supervisory signal. This is achieved with a framework that predicts depth and camera motion, with a loss based on reconstructing a target video frame from temporally adjacent frames. In this context, occlusion relates to parts of a scene that can be observed in the target frame but not in a frame used for image reconstruction. Since the image reconstruction is based on sampling from the adjacent frame, and occluded areas by definition cannot be sampled, r…
Erasing Self-Supervised Learning Backdoor by Cluster Activation Masking
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所 多模态人工智能系统国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Xiaomi Inc.（小米公司）；State Key Laboratory of CNS/ATM, Beihang University（北京航空航天大学 CNS/ATM 国家重点实验室）；College of Computer Science, Inner Mongolia University（内蒙古大学 计算机学院）
Progressive Self-Supervised Learning with Individualized Community Assignment for Brain Network Analysis
Harbin Institute of Technology at Shenzhen（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）；University of Tübingen（图宾根大学）；Max Planck Institute for Intelligent Systems（智能系统马克斯·普朗克研究所）
Point-DAE: Denoising Autoencoders for Self-supervised Point Cloud Learning
Department of Computing, The Hong Kong Polytechnic University（计算系，香港理工大学）；School of Electronic and Information Engineering, South China University of Technology（电子与信息工程学院，华南理工大学）
Nanjing University（南京大学）；Tencent AI Lab（腾讯人工智能实验室）；Shanghai AI Lab（上海人工智能实验室）
Pre-training video transformers on extra large-scale datasets is generally required to achieve premier performance on relatively small datasets. In this paper, we show that video masked autoencoders (VideoMAE) are data-efficient learners for self-supervised video pre-training (SSVP). We are inspired by the recent ImageMAE and propose customized video tube masking with an extremely high ratio. This simple design makes video reconstruction a more challenging self-supervision task, thus encouraging extracting more effective video representations during this pre-training process. We obtain three i…
HySparK: Hybrid Sparse Masking for Large Scale Medical Image Pre-Training
Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）；Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS（中国科学院计算技术研究所智能信息处理重点实验室）
