Coding images for machines with minimal bitrate and strong analysis performance is key to effective edge-cloud systems. Several approaches deploy an image codec and perform analysis on the reconstructed image. Other methods compress intermediate features using entropy models and subsequently perform analysis on the decoded features. Nevertheless, these methods both perform poorly under low-bitrate conditions, as they retain many redundant details or learn over-concentrated symbol distributions. In this paper, we propose a Codebook-based Adaptive Feature Compression framework with Semantic Enhancement, named CAFC-SE. It maps continuous visual features to discrete indices with a codebook at the edge via Vector Quantization (VQ) and selectively transmits them to the cloud. The VQ operation that projects feature vectors onto the nearest visual primitives enables us to preserve more informative visual patterns under low-bitrate conditions. Hence, CAFC-SE is less vulnerable to low-bitrate conditions. Extensive experiments demonstrate the superiority of our method in terms of rate and accuracy.
In VQ based image compression technique has three major steps namely (i) Codebook Design, (ii) VQ Encoding Process and (iii) VQ Decoding Process. The performance of VQ based image compression technique depends upon the constructed codebook. A widely used technique for VQ codebook design is the Linde-Buzo-Gray (LBG) algorithm. However the performance of the standard LBG algorithm is highly dependent on the choice of the initial codebook. In this paper, we have proposed a simple and very effective approach for codebook initialization for LBG algorithm. The simulation results show that the propos…
Recently, Babenko and Lempitsky introduced Additive Quantization (AQ), a generalization of Product Quantization (PQ) where a non-independent set of codebooks is used to compress vectors into small binary codes. Unfortunately, under this scheme encoding cannot be done independently in each codebook, and optimal encoding is an NP-hard problem. In this paper, we observe that PQ and AQ are both compositional quantizers that lie on the extremes of the codebook dependence-independence assumption, and explore an intermediate approach that exploits a hierarchical structure in the codebooks. This resul…
Vector Quantization, VQ is a popular image compression technique with a simple decoding architecture and high compression ratio. Codebook designing is the most essential part in Vector Quantization. LindeBuzoGray, LBG is a traditional method of generation of VQ Codebook which results in lower PSNR value. A Codebook affects the quality of image compression, so the choice of an appropriate codebook is a must. Several optimization techniques have been proposed for global codebook generation to enhance the quality of image compression. In this paper, a novel algorithm called IDE-LBG is proposed wh…
CodeEnhance: A Codebook-Driven Approach for Low-Light Image Enhancement
Computer Vision Institute, College of Computer Science and Software Engineering, Shenzhen University（计算机视觉研究所，计算机科学与软件工程学院，深圳大学）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（社会人工智能与机器人研究所）；Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen University（广东省智能信息处理重点实验室，深圳大学）；School of AI and Advanced Computing, Xi’an Jiaotong-Liverpool University（人工智能与高级计算学院，西安交通大学-利物浦大学）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（大数据系统计算技术国家工程实验室，深圳大学）；Department of Electrical & Computer Engineering, University of Alberta（电气与计算机工程系，阿尔伯塔大学）
Wangxuan Institute of Computer Technology（王选计算机研究所）；Peking University（北京大学）
Visual analytics have played an increasingly critical role in the Internet of Things, where massive visual signals have to be compressed and fed into machines. But facing such big data and constrained bandwidth capacity, existing image/video compression methods lead to very low-quality representations, while existing feature compression techniques fail to support diversified visual analytics applications/tasks with low-bit-rate representations. In this paper, we raise and study the novel problem of supporting multiple machine vision analytics tasks with the compressed visual representation, na…
In this paper, we propose a privacy-preserving image-retrieval scheme using a codebook generated by using a plain-image dataset. Encryption-then-compression (EtC) images, which were proposed for EtC systems, have been used in conventional privacy-preserving image-retrieval schemes, in which a codebook is generated from EtC images uploaded by image owners, and extended SIMPLE descriptors are then calculated as image descriptors by using the codebook. In contrast, in the proposed scheme, a codebook is generated from a dataset independent of uploaded images. The use of an independent dataset enab…
BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebook
The Hong Kong University of Science and Technology（香港科技大学）；City University of Hong Kong（香港城市大学）；University of Electronic Science and Technology of China（电子科技大学）
University of Electronic Science and Technology of China（电子科技大学）；Southern University of Science and Technology（南方科技大学）
Modeling latent variables with priors and hyperpriors is an essential problem in variational image compression. Formally, trade-off between rate and distortion is handled well if priors and hyperpriors precisely describe latent variables. Current practices only adopt univariate priors and process each variable individually. However, we find inter-correlations and intra-correlations exist when observing latent variables in a vectorized perspective. These findings reveal visual redundancies to improve rate-distortion performance and parallel processing ability to speed up compression. This encou…
Compressed Image Generation with Denoising Diffusion Codebook Models
Faculty of Computer Science, Technion -- Israel Institute of Technology, Haifa, Israel（计算机科学系，技术离子理工学院——以色列理工学院，海法，以色列）；Faculty of Electrical and Computer Engineering, Technion -- Israel Institute of Technology, Haifa, Israel（电气与计算机工程系，技术离子理工学院——以色列理工学院，海法，以色列）
A learning-based framework for representation of domain-specific images is proposed where joint compression and denoising can be done using a VQ-based multi-layer network. While it learns to compress the images from a training set, the compression performance is very well generalized on images from a test set. Moreover, when fed with noisy versions of the test set, since it has priors from clean images, the network also efficiently denoises the test images during the reconstruction. The proposed framework is a regularized version of the Residual Quantization (RQ) where at each stage, the quant…
VQCNIR: Clearer Night Image Restoration with Vector-Quantized Codebook
South China University of Technology（华南理工大学）；Pazhou Laboratory（琶洲实验室）；The Hong Kong University of Science and Technology（香港科技大学）；Fujian Normal University（福建师范大学）
The end-to-end learning-based video compression has attracted substantial attentions by paving another way to compress video signals as stacked visual features. This paper proposes an efficient end-to-end deep video codec with jointly optimized compression and enhancement modules (JCEVC). First, we propose a dual-path generative adversarial network (DPEG) to reconstruct video details after compression. An $α$-path facilitates the structure information reconstruction with a large receptive field and multi-frame references, while a $β$-path facilitates the reconstruction of local textures. Both …
In this paper, we propose a Hierarchical Learned Video Compression (HLVC) method with three hierarchical quality layers and a recurrent enhancement network. The frames in the first layer are compressed by an image compression method with the highest quality. Using these frames as references, we propose the Bi-Directional Deep Compression (BDDC) network to compress the second layer with relatively high quality. Then, the third layer frames are compressed with the lowest quality, by the proposed Single Motion Deep Compression (SMDC) network, which adopts a single motion map to estimate the motio…
In this paper, we propose a novel end-to-end feature compression scheme by leveraging the representation and learning capability of deep neural networks, towards intelligent front-end equipped analysis with promising accuracy and efficiency. In particular, the extracted features are compactly coded in an end-to-end manner by optimizing the rate-distortion cost to achieve feature-in-feature representation. In order to further improve the compression performance, we present a latent code level teacher-student enhancement model, which could efficiently transfer the low bit-rate representation int…
Electronics and Telecommunications Research Institute（电子与电信研究院）；Korea Advanced Institute of Science and Technology（韩国科学技术院）；Kyungnam University（庆南大学）
Recently, learned image compression methods have been actively studied. Among them, entropy-minimization based approaches have achieved superior results compared to conventional image codecs such as BPG and JPEG2000. However, the quality enhancement and rate-minimization are conflictively coupled in the process of image compression. That is, maintaining high image quality entails less compression and vice versa. However, by jointly training separate quality enhancement in conjunction with image compression, the coding efficiency can be improved. In this paper, we propose a novel joint learning…
Enhancing Underwater Images via Adaptive Semantic-aware Codebook Learning
School of Information Science and Engineering, Ocean University of China（信息科学与工程学院，中国海洋大学）；Department of Electrical and Computer Engineering, Mississippi State University（电气与计算机工程学院，密苏里州立大学）
Beijing University of Posts and Telecommunications（北京邮电大学）；Huawei（华为）
Vector Quantization (VQ) is an appealing model compression method to obtain a tiny model with less accuracy loss. While methods to obtain better codebooks and codes under fixed clustering dimensionality have been extensively studied, optimizations of the vectors in favour of clustering performance are not carefully considered, especially via the reduction of vector dimensionality. This paper reports our recent progress on the combination of dimensionality compression and vector quantization, proposing a Low-Rank Representation Vector Quantization ($\text{LR}^2\text{VQ}$) method that outperform…
CAPformer: Compression-Aware Pre-trained Transformer for Low-Light Image Enhancement
School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区智能系统工程学院）；Guangdong Provincial Key Laboratory of Fire Science and Technology（广东省火灾科学与技术重点实验室）
Standardizing Generative Face Video Compression using Supplemental Enhancement Information
DAMO Academy, Alibaba Group and Hupan Lab（达摩院、阿里巴巴集团和虎扑实验室）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；School of Electronic Information and Electronic Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电子工程学院）；Bytedance Inc.（字节跳动公司）；Dolby Laboratories, Inc.（杜比实验室）
Peking University（北京大学）；City University of Hong Kong（香港城市大学）
In this paper, we propose a scalable image compression scheme, including the base layer for feature representation and enhancement layer for texture representation. More specifically, the base layer is designed as the deep learning feature for analysis purpose, and it can also be converted to the fine structure with deep feature reconstruction. The enhancement layer, which serves to compress the residuals between the input image and the signals generated from the base layer, aims to faithfully reconstruct the input texture. The proposed scheme can feasibly inherit the advantages of both compre…
Enhancing Vector Quantization with Distributional Matching: A Theoretical and Empirical Study
University of Toronto（多伦多大学）；The Hong Kong University of Science and Technology（香港科学与技术大学）；Boston College（波士顿学院）；Lehigh University（莱特大学）；Southern University of Science and Technology（南方科技大学）
Semantics Disentanglement and Composition for Universal Image Coding with Efficiently LLM Reasoning and Generative Diffusion
Shanghai Jiao Tong University（上海交通大学）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波空间智能与数字衍生关键实验室）；Ningbo Institute of Digital Twin（宁波数字孪生研究院）；Eastern Institute of Technology（东部技术研究院）；University of Science and Technology of China（中国科学技术大学）；Yokohama National University（Yokohama国立大学）
