ROI-based Deep Image Compression with Swin Transformers
School of Engineering Science, Simon Fraser University（西蒙菲莎大学工程科学学院）；School of Microelectronics, Xi’an Jiaotong University（西安交通大学微电子学院）；Google LLC（谷歌有限责任公司）
Encoding the Region Of Interest (ROI) with better quality than the background has many applications including video conferencing systems, video surveillance and object-oriented vision tasks. In this paper, we propose a ROI-based image compression framework with Swin transformers as main building blocks for the autoencoder network. The binary ROI mask is integrated into different layers of the network to provide spatial information guidance. Based on the ROI mask, we can control the relative importance of the ROI and non-ROI by modifying the corresponding Lagrange multiplier $ λ$ for different regions. Experimental results show our model achieves higher ROI PSNR than other methods and modest average PSNR for human evaluation. When tested on models pre-trained with original images, it has superior object detection and instance segmentation performance on the COCO validation dataset.
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；Center of China-Singapore International Joint Research Institute（中国-新加坡国际联合研究机构中心）；School of Information Science and Engineering, Shandong Normal University（山东师范大学信息科学与工程学院）；School of Engineering Science, Simon Fraser University（西蒙弗雷泽大学工程科学学院）；School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Institute of Information Science, Beijing Jiao Tong University（北京交通大学信息科学学院）
We propose a deep learning system for attention-guided dual-layer image compression (AGDL). In the AGDL compression system, an image is encoded into two layers, a base layer and an attention-guided refinement layer. Unlike the existing ROI image compression methods that spend an extra bit budget equally on all pixels in ROI, AGDL employs a CNN module to predict those pixels on and near a saliency sketch within ROI that are critical to perceptual quality. Only the critical pixels are further sampled by compressive sensing (CS) to form a very compact refinement layer. Another novel CNN method is…
University of Würzburg（维尔茨堡大学）；MegaStudyEdu（MegaStudy教育）
Compression plays an important role on the efficient transmission and storage of images and videos through band-limited systems such as streaming services, virtual reality or videogames. However, compression unavoidably leads to artifacts and the loss of the original information, which may severely degrade the visual quality. For these reasons, quality enhancement of compressed images has become a popular research topic. While most state-of-the-art image restoration methods are based on convolutional neural networks, other transformers-based methods such as SwinIR, show impressive performance …
University of Science and Technology of China（中国科学技术大学）
Image compression has raised widespread interest recently due to its significant importance for multimedia storage and transmission. Meanwhile, a reliable image quality assessment (IQA) for compressed images can not only help to verify the performance of various compression algorithms but also help to guide the compression optimization in turn. In this paper, we design a full-reference image quality assessment metric SwinIQA to measure the perceptual quality of compressed images in a learned Swin distance space. It is known that the compression artifacts are usually non-uniformly distributed w…
University of Science and Technology of China（中国科学技术大学）
Compressed Image Super-resolution has achieved great attention in recent years, where images are degraded with compression artifacts and low-resolution artifacts. Since the complex hybrid distortions, it is hard to restore the distorted image with the simple cooperation of super-resolution and compression artifacts removing. In this paper, we take a step forward to propose the Hierarchical Swin Transformer (HST) network to restore the low-resolution compressed image, which jointly captures the hierarchical feature representations and enhances each-scale representation with Swin transformer, re…
Lossy face image compression can degrade the image quality and the utility for the purpose of face recognition. This work investigates the effect of lossy image compression on a state-of-the-art face recognition model, and on multiple face image quality assessment models. The analysis is conducted over a range of specific image target sizes. Four compression types are considered, namely JPEG, JPEG 2000, downscaled PNG, and notably the new JPEG XL format. Frontal color images from the ColorFERET database were used in a Region Of Interest (ROI) variant and a portrait variant. We primarily conclu…
Image restoration is a long-standing low-level vision problem that aims to restore high-quality images from low-quality images (e.g., downscaled, noisy and compressed images). While state-of-the-art image restoration methods are based on convolutional neural networks, few attempts have been made with Transformers which show impressive performance on high-level vision tasks. In this paper, we propose a strong baseline model SwinIR for image restoration based on the Swin Transformer. SwinIR consists of three parts: shallow feature extraction, deep feature extraction and high-quality image recons…
End-to-End Image Compression with Segmentation Guided Dual Coding for Wind Turbines
Wind Power LAB（风电实验室）；Institut de Robòtica i Informàtica Industrial, CSIC-UPC（工业机器人研究所，西班牙国家研究委员会-加泰罗尼亚理工大学）；Department of Photonics Engineering, Technical University of Denmark (DTU)（丹麦技术大学光子工程系）
Neural image compression (NIC) has outperformed traditional image codecs in rate-distortion (R-D) performance. However, it usually requires a dedicated encoder-decoder pair for each point on R-D curve, which greatly hinders its practical deployment. While some recent works have enabled bitrate control via conditional coding, they impose strong prior during training and provide limited flexibility. In this paper we propose Code Editing, a highly flexible coding method for NIC based on semi-amortized inference and adaptive quantization. Our work is a new paradigm for variable bitrate NIC. Furthe…
Graduate School of Science and Engineering, Hosei University（法政大学理工学研究生院）
COVID-19 leads to the high demand for remote interactive systems ever seen. One of the key elements of these systems is video streaming, which requires a very high network bandwidth due to its specific real-time demand, especially with high-resolution video. Existing video compression methods are struggling in the trade-off between video quality and the speed requirement. Addressed that the background information rarely changes in most remote meeting cases, we introduce a Region-Of-Interests (ROI) based video compression framework (named RCLC) that leverages the cutting-edge learning-based and…
Waseda University（早稻田大学）；Waseda Research Institute for Science and Engineering（早稻田大学科学与工程研究院）；Shanghai Jiao Tong University（上海交通大学）
Learned image compression (LIC) methods have exhibited promising progress and superior rate-distortion performance compared with classical image compression standards. Most existing LIC methods are Convolutional Neural Networks-based (CNN-based) or Transformer-based, which have different advantages. Exploiting both advantages is a point worth exploring, which has two challenges: 1) how to effectively fuse the two methods? 2) how to achieve higher performance with a suitable complexity? In this paper, we propose an efficient parallel Transformer-CNN Mixture (TCM) block with a controllable compl…
A Transformer-based Image Compression (TIC) approach is developed which reuses the canonical variational autoencoder (VAE) architecture with paired main and hyper encoder-decoders. Both main and hyper encoders are comprised of a sequence of neural transformation units (NTUs) to analyse and aggregate important information for more compact representation of input image, while the decoders mirror the encoder-side operations to generate pixel-domain image reconstruction from the compressed bitstream. Each NTU is consist of a Swin Transformer Block (STB) and a convolutional layer (Conv) to best emb…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Centre for Artificial Intelligence and Robotics, HKISI_CAS（中国科学院香港创新研究院人工智能与机器人创新中心）
Learned image compression methods have exhibited superior rate-distortion performance than classical image compression standards. Most existing learned image compression models are based on Convolutional Neural Networks (CNNs). Despite great contributions, a main drawback of CNN based model is that its structure is not designed for capturing local redundancy, especially the non-repetitive textures, which severely affects the reconstruction quality. Therefore, how to make full use of both global structure and local texture becomes the core problem for learning-based image compression. Inspired …
Humans do not perceive all parts of a scene with the same resolution, but rather focus on few regions of interest (ROIs). Traditional Object-Based codecs take advantage of this biological intuition, and are capable of non-uniform allocation of bits in favor of salient regions, at the expense of increased distortion the remaining areas: such a strategy allows a boost in perceptual quality under low rate constraints. Recently, several neural codecs have been introduced for video compression, yet they operate uniformly over all spatial locations, lacking the capability of ROI-based processing. In…
A Compact Hybrid Convolution--Frequency State Space Network for Learned Image Compression
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University（西安交通大学人工智能与机器人研究所人机混合增强智能国家重点实验室）
Progressive Learning with Visual Prompt Tuning for Variable-Rate Image Compression
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）；Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies（广东省新型安全智能技术重点实验室）；Huawei Technologies Company Ltd（华为技术有限公司）；Shenzhen University（深圳大学）
MedROI: Codec-Agnostic Region of Interest-Centric Compression for Medical Images
Division of Computer Engineering, Hankuk University of Foreign Studies（韩国外国语大学计算机工程系）；Department of Computer Science and Engineering, Hankuk University of Foreign Studies（韩国外国语大学计算机科学与工程系）
Less Contouring, More Accuracy: Lesion-Guided ROI Deep Learning for Ovarian Ultrasound Classification
Danube Private University (DPU)（多瑙河私立大学）；Austrian Centre for Medical Innovation and Technology (ACMIT)（奥地利医学创新与技术中心）；Medical University of Vienna（维也纳医科大学）；Urmia University of Medical Science（乌尔米亚医科大学）
Gaze tracking is a valuable tool with a broad range of applications in various fields, including medicine, psychology, virtual reality, marketing, and safety. Therefore, it is essential to have gaze tracking software that is cost-efficient and high-performing. Accurately predicting gaze remains a difficult task, particularly in real-world situations where images are affected by motion blur, video compression, and noise. Super-resolution has been shown to improve image quality from a visual perspective. This work examines the usefulness of super-resolution for improving appearance-based gaze tr…
MS-IQA: A Multi-Scale Feature Fusion Network for PET/CT Image Quality Assessment
Harbin Institute of Technology（哈尔滨工业大学）；Nanjing University of Information Science and Technology（南京信息工程大学）；Harbin Medical University（哈尔滨医科大学）；Nanyang Technological University（南洋理工大学）；Sichuan University of Science and Engineering（四川理工学院）
High-Frequency Prior-Driven Adaptive Masking for Accelerating Image Super-Resolution
School of Computer Science and Technology, Harbin Institute of Technology（计算机科学与技术学院，哈尔滨工业大学）；City University of Hong Kong（香港城市大学）；Tianjin University（天津大学）
CFVNet: An End-to-End Cancelable Finger Vein Network for Recognition
School of Cyber Science and Engineering, Southeast University（东南大学网络空间科学与工程学院）；Purple Mountain Laboratories（紫金山实验室）；University of Macao（澳门大学）；The Hong Kong University of Science and Technology（香港科技大学）
Real-World Blind Super-Resolution via Feature Matching with Implicit High-Resolution Priors
School of Informatics, Xiamen University（厦门大学信息学院）；School of Computer Science（计算机科学学院）；Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；SSE, The Chinese University of Hong Kong（香港中文大学（深圳）SSE）
Huazhong University of Science and Technology（华中科技大学）；Essex University（埃塞克斯大学）；Hong Kong University of Science and Technology（香港科技大学）
Deep learning has shown impressive performance in semantic segmentation, but it is still unaffordable for resource-constrained mobile devices. While offloading computation tasks is promising, the high traffic demands overwhelm the limited bandwidth. Existing compression algorithms are not fit for semantic segmentation, as the lack of obvious and concentrated regions of interest (RoIs) forces the adoption of uniform compression strategies, leading to low compression ratios or accuracy. This paper introduces STAC, a DNN-driven compression scheme tailored for edge-assisted semantic video segmenta…
