Traditional video codecs optimized for pixel fidelity collapse at ultra-low bitrates and produce severe artifacts. This failure arises from a fundamental misalignment between pixel accuracy and human perception. We propose a semantic video compression framework named DiSCo that transmits only the most meaningful information while relying on generative priors for detail synthesis. The source video is decomposed into three compact modalities: a textual description, a spatiotemporally degraded video, and optional sketches or poses that respectively capture semantic, appearance, and motion cues. A conditional video diffusion model then reconstructs high-quality, temporally coherent videos from these compact representations. Temporal forward filling, token interleaving, and modality-specific codecs are proposed to improve multimodal generation and modality compactness. Experiments show that our method outperforms baseline semantic and traditional codecs by 2-10X on perceptual metrics at low bitrates.
We present a learned image compression system based on GANs, operating at extremely low bitrates. Our proposed framework combines an encoder, decoder/generator and a multi-scale discriminator, which we train jointly for a generative learned compression objective. The model synthesizes details it cannot afford to store, obtaining visually pleasing results at bitrates where previous methods fail and show strong artifacts. Furthermore, if a semantic label map of the original image is available, our method can fully synthesize unimportant regions in the decoded image such as streets and trees from…
Active Sampling for Ultra-Low-Bit-Rate Video Compression via Conditional Controlled Diffusion
Department of Electrical and Computer Engineering, University of California San Diego（电子与计算机工程系，加州大学圣地亚哥分校）；Department of Electrical and Systems Engineering, University of Pennsylvania（电子与系统工程系，宾夕法尼亚大学）
Generative Video Compression with Adaptive Score Distillation
Communication University of China（中国传媒大学）；University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）
Transferring large amount of high resolution images over limited bandwidth is an important but very challenging task. Compressing images using extremely low bitrates (<0.1 bpp) has been studied but it often results in low quality images of heavy artifacts due to the strong constraint in the number of bits available for the compressed data. It is often said that a picture is worth a thousand words but on the other hand, language is very powerful in capturing the essence of an image using short descriptions. With the recent success of diffusion models for text-to-image generation, we propose a g…
Joint Degradation-Aware Arbitrary-Scale Super-Resolution for Variable-Rate Extreme Image Compression
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University（上海交通大学信息科学与电子工程学院）；Department of Electronic Engineer and Information Science, University of Science and Technology of China（中国科学技术大学电子工程与信息科学系）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能教育部重点实验室）
The University of Sydney（悉尼大学）；Beijing Institute of Technology（北京理工大学）
In this paper, we propose a new deep image compression framework called Complexity and Bitrate Adaptive Network (CBANet), which aims to learn one single network to support variable bitrate coding under different computational complexity constraints. In contrast to the existing state-of-the-art learning based image compression frameworks that only consider the rate-distortion trade-off without introducing any constraint related to the computational complexity, our CBANet considers the trade-off between the rate and distortion under dynamic computational complexity constraints. Specifically, to …
We present TACTIC: Task-Aware Compression Through Intelligent Coding. Our lossy compression model learns based on the rate-distortion-accuracy trade-off for a specific task. By considering what information is important for the follow-on problem, the system trades off visual fidelity for good task performance at a low bitrate. When compared against JPEG at the same bitrate, our approach is able to improve the accuracy of ImageNet subset classification by 4.5%. We also demonstrate the applicability of our approach to other problems, providing a 3.4% accuracy and 4.9% mean IoU improvements in per…
MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Model
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像通信与网络工程研究所）；John Hopcroft Center, Shanghai Jiao Tong University（上海交通大学约翰·霍普克罗夫特中心）；S-Lab, Nanyang Technological University（南洋理工大学S-Lab）
Learned image compression methods generally optimize a rate-distortion loss, trading off improvements in visual distortion for added bitrate. Increasingly, however, compressed imagery is used as an input to deep learning networks for various tasks such as classification, object detection, and superresolution. We propose a recognition-aware learned compression method, which optimizes a rate-distortion loss alongside a task-specific loss, jointly learning compression and recognition networks. We augment a hierarchical autoencoder-based compression network with an EfficientNet recognition model a…
Leveraging Compression to Construct Transferable Bitrate Ladders
Laboratory for Image and Video Engineering (LIVE), The University of Texas at Austin（图像与视频工程实验室（LIVE），德克萨斯大学奥斯汀分校）；Colorado’s Laboratory for Image and Video Engineering (LIVE), University of Colorado Boulder（科罗拉多州图像与视频工程实验室（LIVE），科罗拉多大学波德分校）；Meta Platforms Inc.（Meta平台公司）
Generative Latent Coding for Ultra-Low Bitrate Image and Video Compression
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China（脑启发式智能感知与认知联合实验室，中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）
Compressing images at extremely low bitrates (< 0.1 bpp) has always been a challenging task since the quality of reconstruction significantly reduces due to the strong imposed constraint on the number of bits allocated for the compressed data. With the increasing need to transfer large amounts of images with limited bandwidth, compressing images to very low sizes is a crucial task. However, the existing methods are not effective at extremely low bitrates. To address this need, we propose a novel network called CompressNet which augments a Stacked Autoencoder with a Switch Prediction Network (S…
Group-of-Latents: Perceptual Video Compression at Extreme Bitrates via Masked Latent Generative Modeling
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China（多媒体信息处理国家重点实验室，计算机科学学院，北京大学，北京，中国）；State Key Discipline Laboratory of Wide Band-Gap Semiconductor Technology, School of Microelectronics, Xidian University, Xi'an, China（宽带隙半导体技术重点学科实验室，微电子学院，西安电子科技大学，西安，中国）
In this paper we tackle the problem of stereo image compression, and leverage the fact that the two images have overlapping fields of view to further compress the representations. Our approach leverages state-of-the-art single-image compression autoencoders and enhances the compression with novel parametric skip functions to feed fully differentiable, disparity-warped features at all levels to the encoder/decoder of the second image. Moreover, we model the probabilistic dependence between the image codes using a conditional entropy model. Our experiments show an impressive 30 - 50% reduction i…
SDGIC: A Semantic Disambiguation-Guided Generative Image Compression Method for Ultra-Low Bitrates
School of Information and Communications Engineering, Xi’an Jiaotong University（西安交通大学信息与通信工程学院）；Department of Electrical and Computer Engineering, Faculty of Applied Science, The University of British Columbia（不列颠哥伦比亚大学应用科学学院电气与计算机工程系）
Exploiting Semantic and Pixel Representations for Ultra-Low Bitrate Image Compression
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（人机混合增强智能国家重点实验室，人工智能与机器人研究院，西安交通大学）；School of Information and Telecommunication, Xi’an Jiaotong University（信息与电信学院，西安交通大学）；Department of Computer Science and Software Engineering, The University of Western Australia（计算机科学与软件工程系，西澳大学）
In this work we propose a novel deep learning approach for ultra-low bitrate video compression for video conferencing applications. To address the shortcomings of current video compression paradigms when the available bandwidth is extremely limited, we adopt a model-based approach that employs deep neural networks to encode motion information as keypoint displacement and reconstruct the video signal at the decoder side. The overall system is trained in an end-to-end fashion minimizing a reconstruction error on the encoder output. Objective and subjective quality evaluation experiments demonstr…
Huazhong University of Science and Technology（华中科技大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；University of Calgary（卡尔加里大学）；New York University（纽约大学）
Deep learning techniques have shown promising results in image compression, with competitive bitrate and image reconstruction quality from compressed latent. However, while image compression has progressed towards a higher peak signal-to-noise ratio (PSNR) and fewer bits per pixel (bpp), their robustness to adversarial images has never received deliberation. In this work, we, for the first time, investigate the robustness of image compression systems where imperceptible perturbation of input images can precipitate a significant increase in the bitrate of their compressed latent. To characteriz…
Qualcomm AI Research, Qualcomm Technologies, Inc.（高通技术公司高通人工智能研究院）；University of California Irvine（加州大学欧文分校）；Qualcomm AI Research, Qualcomm Technologies Netherlands B.V.（高通技术荷兰有限公司高通人工智能研究院）
We present PLONQ, a progressive neural image compression scheme which pushes the boundary of variable bitrate compression by allowing quality scalable coding with a single bitstream. In contrast to existing learned variable bitrate solutions which produce separate bitstreams for each quality, it enables easier rate-control and requires less storage. Leveraging the latent scaling based variable bitrate solution, we introduce nested quantization, a method that defines multiple quantization levels with nested quantization grids, and progressively refines all latents from the coarsest to the fines…
Deep image compression systems mainly contain four components: encoder, quantizer, entropy model, and decoder. To optimize these four components, a joint rate-distortion framework was proposed, and many deep neural network-based methods achieved great success in image compression. However, almost all convolutional neural network-based methods treat channel-wise feature maps equally, reducing the flexibility in handling different types of information. In this paper, we propose a channel-level variable quantization network to dynamically allocate more bitrates for significant channels and withdr…
