Just Noticeable Difference for Deep Machine Vision
Nanyang Technological University（南洋理工大学）；Alibaba-NTU Singapore Joint Research Institute（阿里巴巴-南洋理工大学新加坡联合研究院）；Tsinghua University（清华大学）；Beijing Jiao Tong University（北京交通大学）；Beijing Key Laboratory of Advanced Information Science and Network Technology（北京市先进信息科学与网络技术重点实验室）；Alibaba Cloud Business Group（阿里巴巴云业务集团）
As an important perceptual characteristic of the Human Visual System (HVS), the Just Noticeable Difference (JND) has been studied for decades with image and video processing (e.g., perceptual visual signal compression). However, there is little exploration on the existence of JND for the Deep Machine Vision (DMV), although the DMV has made great strides in many machine vision tasks. In this paper, we take an initial attempt, and demonstrate that the DMV has the JND, termed as the DMV-JND. We then propose a JND model for the image classification task in the DMV. It has been discovered that the DMV can tolerate distorted images with average PSNR of only 9.56dB (the lower the better), by generating JND via unsupervised learning with the proposed DMV-JND-NET. In particular, a semantic-guided redundancy assessment strategy is designed to restrain the magnitude and spatial distribution of the DMV-JND. Experimental results on image classification demonstrate that we successfully find the JND for deep machine vision. Our DMV-JND facilitates a possible direction for DMV-oriented image and video compression, watermarking, quality assessment, deep neural network security, and so on.
One way of designing a robust machine learning algorithm is to generate authentic adversarial images which can trick the algorithms as much as possible. In this study, we propose a new method to generate adversarial images which are very similar to true images, yet, these images are discriminated from the original ones and are assigned into another category by the model. The proposed method is based on a popular concept of experimental psychology, called, Just Noticeable Difference. We define Just Noticeable Difference for a machine learning model and generate a least perceptible difference fo…
Just noticeable difference (JND) refers to the maximum visual change that human eyes cannot perceive, and it has a wide range of applications in multimedia systems. However, most existing JND approaches only focus on a single modality, and rarely consider the complementary effects of multimodal information. In this article, we investigate the JND modeling from an end-to-end homologous multimodal perspective, namely hmJND-Net. Specifically, we explore three important visually sensitive modalities, including saliency, depth, and segmentation. To better utilize homologous multimodal information, …
In this study, we introduce a measure for machine perception, inspired by the concept of Just Noticeable Difference (JND) of human perception. Based on this measure, we suggest an adversarial image generation algorithm, which iteratively distorts an image by an additive noise until the model detects the change in the image by outputting a false label. The noise added to the original image is defined as the gradient of the cost function of the model. A novel cost function is defined to explicitly minimize the amount of perturbation applied to the input image while enforcing the perceptual simil…
Nanyang Technological University（南洋理工大学）；Alibaba-NTU Singapore Joint Research Institute（阿里巴巴-南洋理工大学新加坡联合研究院）；Fudan University（复旦大学）；Tsinghua University（清华大学）；Shandong Normal University（山东师范大学）；Beijing Jiao Tong University（北京交通大学）
Significant improvement has been made on just noticeable difference (JND) modelling due to the development of deep neural networks, especially for the recently developed unsupervised-JND generation models. However, they have a major drawback that the generated JND is assessed in the real-world signal domain instead of in the perceptual domain in the human brain. There is an obvious difference when JND is assessed in such two domains since the visual signal in the real world is encoded before it is delivered into the brain with the human visual system (HVS). Hence, we propose an HVS-inspired si…
Compared with traditional machine learning models, deep neural networks perform better, especially in image classification tasks. However, they are vulnerable to adversarial examples. Adding small perturbations on examples causes a good-performance model to misclassify the crafted examples, without category differences in the human eyes, and fools deep models successfully. There are two requirements for generating adversarial examples: the attack success rate and image fidelity metrics. Generally, perturbations are increased to ensure the adversarial examples' high attack success rate; however…
The performance of an edge detector can be improved when assisted with an effective edge map quality measure. Several evaluation methods have been proposed resulting in different performance score for the same candidate edge map. However, an effective measure is the one that can be automated and which correlates with human judgement perceived quality of the edge map. Distance-based edge map measures are widely used for assessment of edge map quality. These methods consider distance and statistical properties of edge pixels to estimate a performance score. The existing methods can be automated;…
Nanyang Technological University（南洋理工大学）；Alibaba-NTU Singapore Joint Research Institute（阿里巴巴-南洋理工大学新加坡联合研究院）；Shandong Normal University（山东师范大学）；Alibaba（阿里巴巴集团）
Just Noticeable Difference (JND) has many applications in multimedia signal processing, especially for visual data processing up to date. It's generally defined as the minimum visual content changes that the human can perspective, which has been studied for decades. However, most of the existing methods only focus on the luminance component of JND modelling and simply regard chrominance components as scaled versions of luminance. In this paper, we propose a JND model to generate the JND by taking the characteristics of full RGB channels into account, termed as the RGB-JND. To this end, an RGB-…
Ningbo University（宁波大学）；City University of Hong Kong（香港城市大学）；Nanyang Technological University（南洋理工大学）
Just noticeable difference (JND) of natural images refers to the maximum pixel intensity change magnitude that typical human visual system (HVS) cannot perceive. Existing efforts on JND estimation mainly dedicate to modeling the diverse masking effects in either/both spatial or/and frequency domains, and then fusing them into an overall JND estimate. In this work, we turn to a dramatically different way to address this problem with a top-down design philosophy. Instead of explicitly formulating and fusing different masking effects in a bottom-up way, the proposed JND estimation model dedicates…
Just Noticeable Difference Modeling for Deep Visual Features
College of Computing and Data Science, Nanyang Technological University, Singapore（新加坡南洋理工大学计算与数据科学学院）；Department of Computer Science and Technology, Harbin Institute of Technology, Harbin, China（哈尔滨工业大学计算机科学与技术学院）；School of Artificial Intelligence, Beijing Normal University, Beijing, China（北京师范大学人工智能学院）；School of Computer Science, Peking University, Beijing, China（北京大学计算机学院）
Just Noticeable Difference for Large Multimodal Models
Institute of Image Communication and Information Processing, Shanghai Jiao Tong University, Shanghai 200240, China（上海交通大学图像通信与信息处理研究所）；Shanghai AI Laboratory, Shanghai 200232, China（上海人工智能实验室）；School of Communication and Electronic Engineering, East China Normal University, Shanghai 200241, China（华东师范大学通信与电子工程学院）；School of Computer Science and Engineering, Nanyang Technological University, Singapore 639798（南洋理工大学计算机科学与工程学院）
Recently, due to the strength of deep convolutional neural networks (CNN), many CNN-based image quality assessment (IQA) models have been studied. However, previous CNN-based IQA models likely have yet to utilize the characteristics of the human visual system (HVS) fully for IQA problems when they simply entrust everything to the CNN, expecting it to learn from a training dataset. However, in this paper, we propose a novel saliency-channel attention residual network based on the just-noticeable-difference (JND) concept for full-reference image quality assessments (FR-IQA). It is referred to as…
Just Noticeable Difference Modeling for Token Compression in Vision-Language-Action Models
The Hong Kong Polytechnic University（香港理工大学）；Nanyang Technological University（南洋理工大学）；Bytedance Inc.（字节跳动公司）；Shandong University（山东大学）；CNRS（法国国家科学研究中心）；CentraleSupelec（中央理工学院）；Université Paris-Saclay（巴黎萨克雷大学）
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
We devise a universal adaptive neural layer to "learn" optimal frequency filter for each image together with the weights of the base neural network that performs some computer vision task. The proposed approach takes the source image in the spatial domain, automatically selects the best frequencies from the frequency domain, and transmits the inverse-transform image to the main neural network. Remarkably, such a simple add-on layer dramatically improves the performance of the main network regardless of its design. We observe that the light networks gain a noticeable boost in the performance me…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
To accelerate MRI, the field of compressed sensing is traditionally concerned with optimizing the image quality after a partial undersampling of the measurable $\textit{k}$-space. In our work, we propose to change the focus from the quality of the reconstructed image to the quality of the downstream image analysis outcome. Specifically, we propose to optimize the patterns according to how well a sought-after pathology could be detected or localized in the reconstructed images. We find the optimal undersampling patterns in $\textit{k}$-space that maximize target value functions of interest in c…
Relative attribute models can compare images in terms of all detected properties or attributes, exhaustively predicting which image is fancier, more natural, and so on without any regard to ordering. However, when humans compare images, certain differences will naturally stick out and come to mind first. These most noticeable differences, or prominent differences, are likely to be described first. In addition, many differences, although present, may not be mentioned at all. In this work, we introduce and model prominent differences, a rich new functionality for comparing images. We collect ins…
Shanghai Jiao Tong University（上海交通大学）；Peng Cheng Laboratory（鹏城实验室）；West Virginia University（西弗吉尼亚大学）；City University of Hong Kong（香港城市大学）
No-reference image quality assessment (NR-IQA) aims to quantify how humans perceive visual distortions of digital images without access to their undistorted references. NR-IQA models are extensively studied in computational vision, and are widely used for performance evaluation and perceptual optimization of man-made vision systems. Here we make one of the first attempts to examine the perceptual robustness of NR-IQA models. Under a Lagrangian formulation, we identify insightful connections of the proposed perceptual attack to previous beautiful ideas in computer vision and machine learning. W…
Fakir Mohan University（法基尔·莫汉大学）；Indus International University（印度河国际大学）
Image based rendering is a fundamental problem in computer vision and graphics. Modern techniques often rely on depth image for the 3D construction. However for most of the existing depth cameras, the large and unpredictable noises can be problematic, which can cause noticeable artifacts in the rendered results. In this paper, we proposed an efficacious method for depth image noise removal that can be applied for most RGBD systems. The proposed solution will benefit many subsequent vision problems such as 3D reconstruction, novel view rendering, object recognition. Our experimental results dem…
In perceptual image coding applications, the main objective is to decrease, as much as possible, Bits Per Pixel (BPP) while avoiding noticeable distortions in the reconstructed image. In this paper, we propose a novel perceptual image coding technique, named Perceptual Color Compression (PCC). PCC is based on a novel model related to Human Visual System (HVS) spectral sensitivity and CIELAB Just Noticeable Color Difference (JNCD). We utilize this modeling to capitalize on the inability of the HVS to perceptually differentiate photons in very similar wavelength bands (e.g., distinguishing very …
DT-JRD: Deep Transformer based Just Recognizable Difference Prediction Model for Video Coding for Machines
National Astronomical Observatories, Chinese Academy of Sciences（中国科学院国家天文台）；Department of Computing and Decision Sciences, Lingnan University（岭南大学计算与决策科学系）
GLIMPSE: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them?
UNC Chapel-Hill（北卡罗来纳大学教堂山分校）；Microsoft（微软）；NUS（新加坡国立大学）；University of Minnesota（明尼苏达大学）；Case Western Reserve University（凯斯西储大学）；Rutgers University（罗格斯大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）
RMIT University（皇家墨尔本理工大学）；University of New South Wales（新南威尔士大学）；Philips Research（飞利浦研究院）；Data61, CSIRO（联邦科学与工业研究组织数据61所）
Recently, Self-Supervised Representation Learning (SSRL) has attracted much attention in the field of computer vision, speech, natural language processing (NLP), and recently, with other types of modalities, including time series from sensors. The popularity of self-supervised learning is driven by the fact that traditional models typically require a huge amount of well-annotated data for training. Acquiring annotated data can be a difficult and costly process. Self-supervised methods have been introduced to improve the efficiency of training data through discriminative pre-training of models …
Yonsei University（延世大学）；University of California, Los Angeles（加州大学洛杉矶分校）
Single-image super-resolution aims to generate a high-resolution version of a low-resolution image, which serves as an essential component in many computer vision applications. This paper investigates the robustness of deep learning-based super-resolution methods against adversarial attacks, which can significantly deteriorate the super-resolved images without noticeable distortion in the attacked low-resolution images. It is demonstrated that state-of-the-art deep super-resolution methods are highly vulnerable to adversarial attacks. Different levels of robustness of different methods are ana…
The University of Pennsylvania（宾夕法尼亚大学）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）
We suggest a new color distance based on two observations. First, perceptual color differences were designed to be used to compare very similar colors. They do not capture human perception for medium and large color differences well. Thresholding was proposed to solve the problem for large color differences, i.e. two totally different colors are always the same distance apart. We show that thresholding alone cannot improve medium color differences. We suggest to alleviate this problem using basic color terms. Second, when a color distance is used for edge detection, many small distances around…
Based on the notion of just noticeable differences (JND), a stair quality function (SQF) was recently proposed to model human perception on JPEG images. Furthermore, a k-means clustering algorithm was adopted to aggregate JND data collected from multiple subjects to generate a single SQF. In this work, we propose a new method to derive the SQF using the Gaussian Mixture Model (GMM). The newly derived SQF can be interpreted as a way to characterize the mean viewer experience. Furthermore, it has a lower information criterion (BIC) value than the previous one, indicating that it offers a better …
Multi-task Just Recognizable Difference for Video Coding for Machines: Database, Model, and Coding Application
School of Electronics and Communication Engineering, Shenzhen Campus, Sun Yat-Sen University（中山大学深圳校区电子与通信工程学院）；National Astronomical Observatories, Chinese Academy of Sciences（中国科学院国家天文台）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
Australian National University（澳大利亚国立大学）；Beijing Institute of Technology（北京理工大学）；Tencent（腾讯）
Images captured in snowy days suffer from noticeable degradation of scene visibility, which degenerates the performance of current vision-based intelligent systems. Removing snow from images thus is an important topic in computer vision. In this paper, we propose a Deep Dense Multi-Scale Network (\textbf{DDMSNet}) for snow removal by exploiting semantic and geometric priors. As images captured in outdoor often share similar scenes and their visibility varies with depth from camera, such semantic and geometric information provides a strong prior for snowy image restoration. We incorporate the s…
Indiana University Bloomington（印第安纳大学伯明顿分校）；Northwestern University（西北大学）；K.J. Somaiya College of Engineering（K.J.索马亚工程学院）
Blind people face a lot of problems in their daily routines. They have to struggle a lot just to do their day-to-day chores. In this paper, we have proposed a system with the objective to help the visually impaired by providing audio aid guiding them to avoid obstacles, which will assist them to move in their surroundings. Object Detection using YOLO will help them detect the nearby objects and Depth Estimation using monocular vision will tell the approximate distance of the detected objects from the user. Despite a higher accuracy, stereo vision has many hardware constraints, which makes mono…
Tsinghua University（清华大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Duke University（杜克大学）；Shanghai Jiao Tong University（上海交通大学）；Harbin Institute of Technology（哈尔滨工业大学）；Tianyijiaotong Technology Ltd.（天宜交通科技有限公司）
Peking University（北京大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft（微软）
We propose Unicoder-VL, a universal encoder that aims to learn joint representations of vision and language in a pre-training manner. Borrow ideas from cross-lingual pre-trained models, such as XLM and Unicoder, both visual and linguistic contents are fed into a multi-layer Transformer for the cross-modal pre-training, where three pre-trained tasks are employed, including Masked Language Modeling (MLM), Masked Object Classification (MOC) and Visual-linguistic Matching (VLM). The first two tasks learn context-aware representations for input tokens based on linguistic and visual contents jointly…
Internet of Things (IoT) has catapulted human ability to control our environments through ubiquitous sensing, communication, computation, and actuation. Over the past few years, IoT has joined forces with Machine Learning (ML) to embed deep intelligence at the far edge. TinyML (Tiny Machine Learning) has enabled the deployment of ML models for embedded vision on extremely lean edge hardware, bringing the power of IoT and ML together. However, TinyML powered embedded vision applications are still in a nascent stage, and they are just starting to scale to widespread real-world IoT deployment. To…
