The past few years have witnessed fast development in video quality enhancement via deep learning. Existing methods mainly focus on enhancing the objective quality of compressed video while ignoring its perceptual quality. In this paper, we focus on enhancing the perceptual quality of compressed video. Our main observation is that enhancing the perceptual quality mostly relies on recovering high-frequency sub-bands in wavelet domain. Accordingly, we propose a novel generative adversarial network (GAN) based on multi-level wavelet packet transform (WPT) to enhance the perceptual quality of compressed video, which is called multi-level wavelet-based GAN (MW-GAN). In MW-GAN, we first apply motion compensation with a pyramid architecture to obtain temporal information. Then, we propose a wavelet reconstruction network with wavelet-dense residual blocks (WDRB) to recover the high-frequency details. In addition, the adversarial loss of MW-GAN is added via WPT to further encourage high-frequency details recovery for video frames. Experimental results demonstrate the superiority of our method.
Imperial College London（帝国理工学院）；ETH Zurich（苏黎世联邦理工学院）；Beihang University（北京航空航天大学）
In single image super-resolution (SISR), given a low-resolution (LR) image, one wishes to find a high-resolution (HR) version of it which is both accurate and photo-realistic. Recently, it has been shown that there exists a fundamental tradeoff between low distortion and high perceptual quality, and the generative adversarial network (GAN) is demonstrated to approach the perception-distortion (PD) bound effectively. In this paper, we propose a novel method based on wavelet domain style transfer (WDST), which achieves a better PD tradeoff than the GAN based methods. Specifically, we propose to …
Xidian University（西安电子科技大学）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）；Northwestern Polytechnical University（西北工业大学）
In this paper, we propose a deformable convolution-based generative adversarial network (DCNGAN) for perceptual quality enhancement of compressed videos. DCNGAN is also adaptive to the quantization parameters (QPs). Compared with optical flows, deformable convolutions are more effective and efficient to align frames. Deformable convolutions can operate on multiple frames, thus leveraging more temporal information, which is beneficial for enhancing the perceptual quality of compressed videos. Instead of aligning frames in a pairwise manner, the deformable convolution can process multiple frames…
Xidian University（西安电子科技大学）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）；BBC Research & Development（英国广播公司研发部）；Northwestern Polytechnical University（西北工业大学）
In this paper we propose a generative adversarial network (GAN) framework to enhance the perceptual quality of compressed videos. Our framework includes attention and adaptation to different quantization parameters (QPs) in a single model. The attention module exploits global receptive fields that can capture and align long-range correlations between consecutive frames, which can be beneficial for enhancing perceptual quality of videos. The frame to be enhanced is fed into the deep network together with its neighboring frames, and in the first stage features at different depths are extracted. …
ETH Zürich（苏黎世联邦理工学院）；Julius Maximilian University of Würzburg（维尔茨堡朱利叶斯·马克西米利安大学）；KU Leuven（荷语鲁汶大学）
This paper proposes a Perceptual Learned Video Compression (PLVC) approach with recurrent conditional GAN. We employ the recurrent auto-encoder-based compression network as the generator, and most importantly, we propose a recurrent conditional discriminator, which judges on raw vs. compressed video conditioned on both spatial and temporal features, including the latent representation, temporal motion and hidden states in recurrent cells. This way, the adversarial training pushes the generated video to be not only spatially photo-realistic but also temporally consistent with the groundtruth an…
Image inpainting aims to complete the missing or corrupted regions of images with realistic contents. The prevalent approaches adopt a hybrid objective of reconstruction and perceptual quality by using generative adversarial networks. However, the reconstruction loss and adversarial loss focus on synthesizing contents of different frequencies and simply applying them together often leads to inter-frequency conflicts and compromised inpainting. This paper presents WaveFill, a wavelet-based inpainting network that decomposes images into multiple frequency bands and fills the missing regions in e…
Compressed sensing (CS) using overcomplete wavelet dictionaries has been a well-investigated topic in the recent times for image and vision applications. In this paper, different overcomplete wavelet transforms have been studied to estimate the best transform. Performance evaluations are carried out for different overcomplete wavelet transforms from highly undersampled and inaccurate measurements for the recovery of images in frequency as well as physical domains.
Remarkable achievements have been attained with Generative Adversarial Networks (GANs) in image-to-image translation. However, due to a tremendous amount of parameters, state-of-the-art GANs usually suffer from low efficiency and bulky memory usage. To tackle this challenge, firstly, this paper investigates GANs performance from a frequency perspective. The results show that GANs, especially small GANs lack the ability to generate high-quality high frequency information. To address this problem, we propose a novel knowledge distillation method referred to as wavelet knowledge distillation. Ins…
Disruptive Attacks on Face Swapping via Low-Frequency Perceptual Perturbations
Department of Mathematics and Artificial Intelligence（数学与人工智能系）；Shandong Artificial Intelligence Institute（山东省人工智能研究所）；Qilu University of Technology（齐鲁大学）
Spatial resolution adaptation is a technique which has often been employed in video compression to enhance coding efficiency. This approach encodes a lower resolution version of the input video and reconstructs the original resolution during decoding. Instead of using conventional up-sampling filters, recent work has employed advanced super-resolution methods based on convolutional neural networks (CNNs) to further improve reconstruction quality. These approaches are usually trained to minimise pixel-based losses such as Mean-Squared Error (MSE), despite the fact that this type of loss metric …
Compressed sensing has empowered quality image reconstruction with fewer data samples than previously though possible. These techniques rely on a sparsifying linear transformation. The Daubechies wavelet transform is a common sparsifying transformation used for this purpose. In this work, we take advantage of the structure of this wavelet transform and identify an affine transformation that increases the sparsity of the result. After inclusion of this affine transformation, we modify the resulting optimization problem to comply with the form of the Basis Pursuit Denoising problem. Finally, we …
University of Trento（特伦托大学）；Technische Universität Berlin（柏林工业大学）
To reduce the storage requirements, remote sensing (RS) images are usually stored in compressed format. Existing scene classification approaches using deep neural networks (DNNs) require to fully decompress the images, which is a computationally demanding task in operational applications. To address this issue, in this paper we propose a novel approach to achieve scene classification in JPEG 2000 compressed RS images. The proposed approach consists of two main steps: i) approximation of the finer resolution sub-bands of reversible biorthogonal wavelet filters used in JPEG 2000; and ii) charact…
Shandong Normal University（山东师范大学）；Nanyang Technological University（南洋理工大学）
Recently, learned image compression schemes have achieved remarkable improvements in image fidelity (e.g., PSNR and MS-SSIM) compared to conventional hybrid image coding ones due to their high-efficiency non-linear transform, end-to-end optimization frameworks, etc. However, few of them take the Just Noticeable Difference (JND) characteristic of the Human Visual System (HVS) into account and optimize learned image compression towards perceptual quality. To address this issue, a JND-based perceptual quality loss is proposed. Considering that the amounts of distortion in the compressed image at …
Image based Deep Feature Quality Metrics (DFQMs) have been shown to better correlate with subjective perceptual scores over traditional metrics. The fundamental focus of these DFQMs is to exploit internal representations from a large scale classification network as the metric feature space. Previously, no attention has been given to the problem of identifying which layers are most perceptually relevant. In this paper we present a new method for selecting perceptually relevant layers from such a network, based on a neuroscience interpretation of layer behaviour. The selected layers are treated …
High frame rates have been known to enhance the perceived visual quality of specific video content. However, the lack of investigation of high frame rates has restricted the expansion of this research field particularly in the context of full-high-definition (FHD) and 4K ultra-high-definition video formats. This study involves a subjective and objective quality assessment of compressed FHD videos. First, we compress the FHD videos by employing high-efficiency video coding, and VP9 at five quantization parameter levels for multiple frame rates, i.e., 15fps, 30fps, and 60fps. The FHD videos are …
RBFIM: Perceptual Quality Assessment for Compressed Point Clouds Using Radial Basis Function Interpolation
School of Electronics and Information, Northwestern Polytechnical University（电子与信息学院，西北工业大学）；School of Engineering, Royal Melbourne Institute of Technology（工程学院，皇家墨尔本理工学院）；Department of Computer Science, City University of Hong Kong（计算机科学系，香港城市大学）；School of Telecommunication Engineering, Xidian University（电信工程学院，西安电子科技大学）
University of Science and Technology of China（中国科学技术大学）
Image compression has raised widespread interest recently due to its significant importance for multimedia storage and transmission. Meanwhile, a reliable image quality assessment (IQA) for compressed images can not only help to verify the performance of various compression algorithms but also help to guide the compression optimization in turn. In this paper, we design a full-reference image quality assessment metric SwinIQA to measure the perceptual quality of compressed images in a learned Swin distance space. It is known that the compression artifacts are usually non-uniformly distributed w…
The scattering transform network (STN), which has a similar structure as that of a popular convolutional neural network except its use of predefined convolution filters and a small number of layers, can generates a robust representation of an input signal relative to small deformations. We propose a novel Monogenic Wavelet Scattering Network (MWSN) for 2D texture image classification through a cascade of monogenic wavelet filtering with nonlinear modulus and averaging operators by replacing the 2D Morlet wavelet filtering in the standard STN. Our MWSN can extract useful hierarchical and direct…
Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Walton Institute for Information and Communication Systems Science（沃尔顿信息与通信系统科学研究所）；Waterford Institute of Technology（沃特福德理工学院）；University of Johannesburg（约翰内斯堡大学）；University of Sindh（信德大学）；Sungkyunkwan University（成均馆大学）
Deep learning based single image super resolution (SISR) algorithms has revolutionized the overall diagnosis framework by continually improving the architectural components and training strategies associated with convolutional neural networks (CNN) on low-resolution images. However, existing work lacks in two ways: i) the SR output produced exhibits poor texture details, and often produce blurred edges, ii) most of the models have been developed for a single modality, hence, require modification to adapt to a new one. This work addresses (i) by proposing generative adversarial network (GAN) wi…
Optimizing 4D Lookup Table for Low-light Video Enhancement via Wavelet Priori
College of Computer Science and Engineering, Chongqing University of Technology（重庆理工大学计算机科学与工程学院）；Chinese University of Hong Kong（香港中文大学）；College of Computer Science, Chongqing University（重庆大学计算机学院）
Perceptual Tone Mapping Model for High Dynamic Range Imaging
State Key Laboratory of Modern Optical Instrumentation, Zhejiang University（浙江大学现代光学仪器国家重点实验室）；Leeds Institute of Textiles and Colour, University of Leeds（利兹大学利兹纺织与色彩学院）
Shandong University（山东大学）；De Montfort University（德蒙福特大学）；Qingdao University（青岛大学）；City University of Hong Kong（香港城市大学）
In rate-distortion optimization, the encoder settings are determined by maximizing a reconstruction quality measure subject to a constraint on the bit rate. One of the main challenges of this approach is to define a quality measure that can be computed with low computational cost and which correlates well with perceptual quality. While several quality measures that fulfil these two criteria have been developed for images and video, no such one exists for 3D point clouds. We address this limitation for the video-based point cloud compression (V-PCC) standard by proposing a linear perceptual qua…
Aliraqia University（伊拉克大学）；University of Technology（科技大学）；Kyiv National University of Construction and Architecture（基辅国立建筑与建筑工程大学）
The Internet has turned the entire world into a small village;this is because it has made it possible to share millions of images and videos. However, sending and receiving a huge amount of data is considered to be a main challenge. To address this issue, a new algorithm is required to reduce image bits and represent the data in a compressed form. Nevertheless, image compression is an important application for transferring large files and images. This requires appropriate and efficient transfers in this field to achieve the task and reach the best results. In this work, we propose a new algori…
With the advent of multi-coil imaging and compressed sensing, a number of model based reconstruction algorithms have been created. They incorporate a multitude of different regularization functions based on physics, observed phenomenology, and heuristics. Moreover, several iterative methods exist that attempt to simultaneously estimate the sensitivity maps and the image. In this manuscript, we present a generalization of several existing iterative model based algorithms. We devise a calibrationless instance of this generalization that only incorporates regularization terms based on physics and…
Peking University（北京大学）；Hefei University of Technology（合肥工业大学）；Peng Cheng Laboratory（鹏城实验室）
Scaling and lossy coding are widely used in video transmission and storage. Previous methods for enhancing the resolution of such videos often ignore the inherent interference between resolution loss and compression artifacts, which compromises perceptual video quality. To address this problem, we present a mixed-resolution coding framework, which cooperates with a reference-based DCNN. In this novel coding chain, the reference-based DCNN learns the direct mapping from low-resolution (LR) compressed video to their high-resolution (HR) clean version at the decoder side. We further improve recon…
Motion estimation is one of the important procedures in the all video encoders. Most of the complexity of the video coder depends on the complexity of the motion estimation step. The original motion estimation algorithm has a remarkable complexity and therefore many improvements were proposed to enhance the crude version of the motion estimation. The basic idea of many of these works were to optimize some distortion function for mean squared error (MSE) or sum of absolute difference (SAD) in block matching But it is shown that these metrics do not conclude the quality as it is, on the other ha…
