CAPformer: Compression-Aware Pre-trained Transformer for Low-Light Image Enhancement
School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区智能系统工程学院）；Guangdong Provincial Key Laboratory of Fire Science and Technology（广东省火灾科学与技术重点实验室）
Low-Light Image Enhancement (LLIE) has advanced with the surge in phone photography demand, yet many existing methods neglect compression, a crucial concern for resource-constrained phone photography. Most LLIE methods overlook this, hindering their effectiveness. In this study, we investigate the effects of JPEG compression on low-light images and reveal substantial information loss caused by JPEG due to widespread low pixel values in dark areas. Hence, we propose the Compression-Aware Pre-trained Transformer (CAPformer), employing a novel pre-training strategy to learn lossless information from uncompressed low-light images. Additionally, the proposed Brightness-Guided Self-Attention (BGSA) mechanism enhances rational information gathering. Experiments demonstrate the superiority of our approach in mitigating compression effects on LLIE, showcasing its potential for improving LLIE in resource-constrained scenarios.
In this paper, we propose a Hierarchical Learned Video Compression (HLVC) method with three hierarchical quality layers and a recurrent enhancement network. The frames in the first layer are compressed by an image compression method with the highest quality. Using these frames as references, we propose the Bi-Directional Deep Compression (BDDC) network to compress the second layer with relatively high quality. Then, the third layer frames are compressed with the lowest quality, by the proposed Single Motion Deep Compression (SMDC) network, which adopts a single motion map to estimate the motio…
Electronics and Telecommunications Research Institute（电子与电信研究院）；Korea Advanced Institute of Science and Technology（韩国科学技术院）；Kyungnam University（庆南大学）
Recently, learned image compression methods have been actively studied. Among them, entropy-minimization based approaches have achieved superior results compared to conventional image codecs such as BPG and JPEG2000. However, the quality enhancement and rate-minimization are conflictively coupled in the process of image compression. That is, maintaining high image quality entails less compression and vice versa. However, by jointly training separate quality enhancement in conjunction with image compression, the coding efficiency can be improved. In this paper, we propose a novel joint learning…
Interleaved Block-based Learned Image Compression with Feature Enhancement and Quantization Error Compensation
School of software, Shandong University（山东大学软件学院）；School of Control Science and Engineering, Shandong University（山东大学控制科学与工程学院）；School of Engineering and Sustainable Development, De Montfort University（德蒙福特大学工程与可持续发展学院）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；School of Telecommunications Engineering, Xidian University（西安电子科技大学通信工程学院）
Lossy image compression is pervasively conducted to save communication bandwidth, resulting in undesirable compression artifacts. Recently, extensive approaches have been proposed to reduce image compression artifacts at the decoder side; however, they require a series of architecture-identical models to process images with different quality, which are inefficient and resource-consuming. Besides, it is common in practice that compressed images are with unknown quality and it is intractable for existing approaches to select a suitable model for blind quality enhancement. In this paper, we propo…
University of Florida（佛罗里达大学）；SenseBrain Technology（感脑科技）
While humans can effortlessly transform complex visual scenes into simple words and the other way around by leveraging their high-level understanding of the content, conventional or the more recent learned image compression codecs do not seem to utilize the semantic meanings of visual content to their full potential. Moreover, they focus mostly on rate-distortion and tend to underperform in perception quality especially in low bitrate regime, and often disregard the performance of downstream computer vision algorithms, which is a fast-growing consumer group of compressed images in addition to …
Aliraqia University（伊拉克大学）；University of Technology（科技大学）；Kyiv National University of Construction and Architecture（基辅国立建筑与建筑工程大学）
The Internet has turned the entire world into a small village;this is because it has made it possible to share millions of images and videos. However, sending and receiving a huge amount of data is considered to be a main challenge. To address this issue, a new algorithm is required to reduce image bits and represent the data in a compressed form. Nevertheless, image compression is an important application for transferring large files and images. This requires appropriate and efficient transfers in this field to achieve the task and reach the best results. In this work, we propose a new algori…
City University of Hong Kong（香港城市大学）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）
It plays a fundamental role to compactly represent the visual information towards the optimization of the ultimate utility in myriad visual data centered applications. With numerous approaches proposed to efficiently compress the texture and visual features serving human visual perception and machine intelligence respectively, much less work has been dedicated to studying the interactions between them. Here we investigate the integration of feature and texture compression, and show that a universal and collaborative visual information representation can be achieved in a hierarchical way. In pa…
Beihang University（北京航空航天大学）；Shanghai Jiao Tong University（上海交通大学）；The University of Sydney（悉尼大学）
Image compression is a widely used technique to reduce the spatial redundancy in images. Recently, learning based image compression has achieved significant progress by using the powerful representation ability from neural networks. However, the current state-of-the-art learning based image compression methods suffer from the huge computational cost, which limits their capacity for practical applications. In this paper, we propose a unified framework called Efficient Deep Image Compression (EDIC) based on three new technologies, including a channel attention module, a Gaussian mixture model an…
We present a CNN-based predictive lossless compression scheme for raw color mosaic images of digital cameras. This specialized application problem was previously understudied but it is now becoming increasingly important, because modern CNN methods for image restoration tasks (e.g., superresolution, low lighting enhancement, deblurring), must operate on original raw mosaic images to obtain the best possible results. The key innovation of this paper is a high-order nonlinear CNN predictor of spatial-spectral mosaic patterns. The deep learning prediction can model highly complex sample dependenc…
Peking University（北京大学）；City University of Hong Kong（香港城市大学）
In this paper, we propose a scalable image compression scheme, including the base layer for feature representation and enhancement layer for texture representation. More specifically, the base layer is designed as the deep learning feature for analysis purpose, and it can also be converted to the fine structure with deep feature reconstruction. The enhancement layer, which serves to compress the residuals between the input image and the signals generated from the base layer, aims to faithfully reconstruct the input texture. The proposed scheme can feasibly inherit the advantages of both compre…
University of Electronic Science and Technology of China（电子科技大学）；Kuaishou（快手）
In this paper, we propose a luminance-guided chrominance image enhancement convolutional neural network for HEVC intra coding. Specifically, we firstly develop a gated recursive asymmetric-convolution block to restore each degraded chrominance image, which generates an intermediate output. Then, guided by the luminance image, the quality of this intermediate output is further improved, which finally produces the high-quality chrominance image. When our proposed method is adopted in the compression of color images with HEVC intra coding, it achieves 28.96% and 16.74% BD-rate gains over HEVC for…
Resolving Blind Inverse Problems under Dynamic Range Compression via Structured Forward Operator Modeling
Department of XXX, University of YYY, Location, Country（XXX系，YYY大学，地点，国家）；School of ZZZ, Institute of WWW, Location, Country（ZZZ学院，WWW研究所，地点，国家）；ShanghaiTech University（上海科技大学）；Shanghai Jiao Tong University（上海交通大学）
Simon Fraser University（西蒙弗雷泽大学）；Google Inc.（谷歌公司）；Tencent Technologies（腾讯科技）
Recently it has been shown that deep learning-based image compression has shown the potential to outperform traditional codecs. However, most existing methods train multiple networks for multiple bit rates, which increases the implementation complexity. In this paper, we propose a variable-rate image compression framework, which employs more Generalized Divisive Normalization (GDN) layers than previous GDN-based methods. Novel GDN-based residual sub-networks are also developed in the encoder and decoder networks. Our scheme also uses a stochastic rounding-based scalable quantization. To furthe…
With the development of human communications the usage of Visual Communications has also increased. The advancement of image compression methods is one of the main reasons for the enhancement. This paper first presents main modes of image compression methods such as JPEG and JPEG2000 without mathematical details. Also, the paper describes gradient Haar wavelet transforms in order to construct a preliminary image compression algorithm. Then, a new image compression method is proposed based on the preliminary image compression algorithm that can improve standards of image compression. The new me…
The end-to-end learning-based video compression has attracted substantial attentions by paving another way to compress video signals as stacked visual features. This paper proposes an efficient end-to-end deep video codec with jointly optimized compression and enhancement modules (JCEVC). First, we propose a dual-path generative adversarial network (DPEG) to reconstruct video details after compression. An $α$-path facilitates the structure information reconstruction with a large receptive field and multi-frame references, while a $β$-path facilitates the reconstruction of local textures. Both …
In this paper, we propose a novel end-to-end feature compression scheme by leveraging the representation and learning capability of deep neural networks, towards intelligent front-end equipped analysis with promising accuracy and efficiency. In particular, the extracted features are compactly coded in an end-to-end manner by optimizing the rate-distortion cost to achieve feature-in-feature representation. In order to further improve the compression performance, we present a latent code level teacher-student enhancement model, which could efficiently transfer the low bit-rate representation int…
Most video super-resolution methods focus on restoring high-resolution video frames from low-resolution videos without taking into account compression. However, most videos on the web or mobile devices are compressed, and the compression can be severe when the bandwidth is limited. In this paper, we propose a new compression-informed video super-resolution model to restore high-resolution content without introducing artifacts caused by compression. The proposed model consists of three modules for video super-resolution: bi-directional recurrent warping, detail-preserving flow estimation, and L…
Simon Fraser University（西蒙弗雷泽大学）；Google Inc.（谷歌公司）；Tencent Technologies（腾讯科技）
Recently deep learning-based image compression has shown the potential to outperform traditional codecs. However, most existing methods train multiple networks for multiple bit rates, which increase the implementation complexity. In this paper, we propose a new variable-rate image compression framework, which employs generalized octave convolutions (GoConv) and generalized octave transposed-convolutions (GoTConv) with built-in generalized divisive normalization (GDN) and inverse GDN (IGDN) layers. Novel GoConv- and GoTConv-based residual blocks are also developed in the encoder and decoder net…
FaSDiff: Balancing Perception and Semantics in Face Compression via Stable Diffusion Priors
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Research Center of Artificial Intelligence, Pengcheng Laboratory（人工智能研究中心，鹏城实验室）；Harbin Institute of Technology（哈尔滨工业大学）；Huawei Manufacturing（华为制造）
This paper proposes a new light-weight convolutional neural network (5k parameters) for non-uniform illumination image enhancement to handle color, exposure, contrast, noise and artifacts, etc., simultaneously and effectively. More concretely, the input image is first enhanced using Retinex model from dual different aspects (enhancing under-exposure and suppressing over-exposure), respectively. Then, these two enhanced results and the original image are fused to obtain an image with satisfactory brightness, contrast and details. Finally, the extra noise and compression artifacts are removed to…
In learning-based approaches to image compression, codecs are developed by optimizing a computational model to minimize a rate-distortion objective. Currently, the most effective learned image codecs take the form of an entropy-constrained autoencoder with an entropy model that uses both forward and backward adaptation. Forward adaptation makes use of side information and can be efficiently integrated into a deep neural network. In contrast, backward adaptation typically makes predictions based on the causal context of each symbol, which requires serial processing that prevents efficient GPU /…
Recently, deep convolutional neural networks (CNNs) have been demonstrated remarkable progress on single image super-resolution. However, as the depth and width of the networks increase, CNN-based super-resolution methods have been faced with the challenges of computational complexity and memory consumption in practice. In order to solve the above questions, we propose a deep but compact convolutional network to directly reconstruct the high resolution image from the original low resolution image. In general, the proposed model consists of three parts, which are feature extraction block, stack…
Hierarchical Frequency-based Upsampling and Refining for Compressed Video Quality Enhancement
Hangzhou Dianzi University（杭州电子科技大学）；State Key Laboratory of Media Convergence Production Technology and Systems（媒体融合生产技术与系统国家重点实验室）；Xinhua Zhiyun Technology Co., Ltd.（新华智云科技有限公司）
Standardizing Generative Face Video Compression using Supplemental Enhancement Information
DAMO Academy, Alibaba Group and Hupan Lab（达摩院、阿里巴巴集团和虎扑实验室）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；School of Electronic Information and Electronic Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电子工程学院）；Bytedance Inc.（字节跳动公司）；Dolby Laboratories, Inc.（杜比实验室）
Simon Fraser University（西蒙弗雷泽大学）；Google Inc.（谷歌公司）
Deep learning has revolutionized many computer vision fields in the last few years, including learning-based image compression. In this paper, we propose a deep semantic segmentation-based layered image compression (DSSLIC) framework in which the semantic segmentation map of the input image is obtained and encoded as the base layer of the bit-stream. A compact representation of the input image is also generated and encoded as the first enhancement layer. The segmentation map and the compact version of the image are then employed to obtain a coarse reconstruction of the image. The residual betw…
GFix: Perceptually Enhanced Gaussian Splatting Video Compression
Visual Information Laboratory, University of Bristol（布里斯托大学视觉信息实验室）；University of Bristol（布里斯托大学）；School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；Visionular Inc.（Visionular公司）
Zhejiang University（浙江大学）；University of Electronic Science and Technology of China（电子科技大学）
The data storage has been one of the bottlenecks in surveillance systems. The conventional video compression algorithms such as H.264 and H.265 do not fully utilize the low information density characteristic of the surveillance video. In this paper, we propose a video compression method that extracts and compresses the foreground and background of the video separately. The compression ratio is greatly improved by sharing background information among multiple adjacent frames through an adaptive background updating and interpolation module. Besides, we present two different schemes to compress t…
High Efficiency Wiener Filter-based Point Cloud Quality Enhancement for MPEG G-PCC
School of Control Science and Engineering, Shandong University（山东大学控制科学与工程学院）；School of Computer and Control Engineering, Yantai University（烟台大学计算机与控制工程学院）；Shanghai Institute for Advanced Communication and Data Science, Shanghai University（上海大学上海先进通信与数据科学研究院）
Nanjing University（南京大学）；Shanghai Jiao Tong University（上海交通大学）；Hikvision Research Institute（海康威视研究院）
Networked video applications, e.g., video conferencing, often suffer from poor visual quality due to unexpected network fluctuation and limited bandwidth. In this paper, we have developed a Quality Enhancement Network (QENet) to reduce the video compression artifacts, leveraging the spatial and temporal priors generated by respective multi-scale convolutions spatially and warped temporal predictions in a recurrent fashion temporally. We have integrated this QENet as a standard-alone post-processing subsystem to the High Efficiency Video Coding (HEVC) compliant decoder. Experimental results sho…
Many deep learning based video compression artifact removal algorithms have been proposed to recover high-quality videos from low-quality compressed videos. Recently, methods were proposed to mine spatiotemporal information via utilizing multiple neighboring frames as reference frames. However, these post-processing methods take advantage of adjacent frames directly, but neglect the information of the video itself, which can be exploited. In this paper, we propose an effective reference frame proposal strategy to boost the performance of the existing multi-frame approaches. Besides, we introdu…
Curvelet-Based Frequency-Aware Feature Enhancement for Deepfake Detection
Department of Computer Science, College of Science, University of Zakho（扎赫大学计算机科学系）；PRIME Lab, Scientific Research Center, University of Zakho（扎赫大学PRIME实验室）
University of Electronic Science and Technology of China（电子科技大学）；SZ DJI Technology Co., Ltd.（深圳市大疆创新科技有限公司）
Convolution neural network (CNN) based methods offer effective solutions for enhancing the quality of compressed image and video. However, these methods ignore using the raw data to enhance the quality. In this paper, we adopt the raw data in the quality enhancement for the HEVC intra-coded image by proposing an online learning-based method. When quality enhancement is demanded, we online train our proposed model at encoder side and then use the parameters to update the model of decoder side. This method not only improves model performance, but also makes one model adoptable to multiple coding…
Transformer architecture has gained popularity due to its ability to scale with large dataset. Consequently, there is a need to reduce the model size and latency, especially for on-device deployment. We focus on vision transformer proposed for image recognition task (Dosovitskiy et al., 2021), and explore the application of different compression techniques such as low rank approximation and pruning for this purpose. Specifically, we investigate a structured pruning method proposed recently in Zhu et al. (2021) and find that mostly feedforward blocks are pruned with this approach, that too, wit…
