PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models
Tsinghua University（清华大学）；OpenGVLab（开放通用实验室）；Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；Nanjing University（南京大学）；The Chinese University of Hong Kong（香港中文大学）
Chenyu Yang, Xuan Dong, Xizhou Zhu, Weijie Su, Jiahao Wang, Hao Tian, Zhe Chen, Wenhai Wang, Lewei Lu, Jifeng Dai
Large Vision-Language Models (VLMs) have been extended to understand both images and videos. Visual token compression is leveraged to reduce the considerable token length of visual inputs. To meet the needs of different tasks, existing high-performance models usually process images and videos separately with different token compression strategies, limiting the capabilities of combining images and videos. To this end, we extend each image into a "static" video and introduce a unified token compression strategy called Progressive Visual Token Compression (PVC), where the tokens of each frame are progressively encoded and adaptively compressed to supplement the information not extracted from previous frames. Video tokens are efficiently compressed with exploiting the inherent temporal redundancy. Images are repeated as static videos, and the spatial details can be gradually supplemented in multiple frames. PVC unifies the token compressing of images and videos. With a limited number of tokens per frame (64 tokens by default), spatial details and temporal changes can still be preserved. Experiments show that our model achieves state-of-the-art performance across various video understanding benchmarks, including long video tasks and fine-grained short video tasks. Meanwhile, our unified token compression strategy incurs no performance loss on image benchmarks, particularly in detail-sensitive tasks.
LLaVA-UHD v3: Progressive Visual Compression for Efficient Native-Resolution Encoding in MLLMs
Tsinghua University（清华大学）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航天信息研究所）；School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉学科学院）
Efficient Multi-modal Large Language Models via Progressive Consistency Distillation
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Duke University（杜克大学）；The University of Hong Kong（香港大学）；Peking University（北京大学）；University of Chicago（芝加哥大学）；Sun Yat-sen University（中山大学）
The University of Tokyo（东京大学）；National Institute of Informatics（国立信息学研究所）
We push forward neural network compression research by exploiting a novel challenging task of large-scale conditional generative adversarial networks (GANs) compression. To this end, we propose a gradually shrinking GAN (PPCD-GAN) by introducing progressive pruning residual block (PP-Res) and class-aware distillation. The PP-Res is an extension of the conventional residual block where each convolutional layer is followed by a learnable mask layer to progressively prune network parameters as training proceeds. The class-aware distillation, on the other hand, enhances the stability of training b…
We study the design of deep architectures for lossy image compression. We present two architectural recipes in the context of multi-stage progressive encoders and empirically demonstrate their importance on compression performance. Specifically, we show that: (a) predicting the original image data from residuals in a multi-stage progressive architecture facilitates learning and leads to improved performance at approximating the original content and (b) learning to inpaint (from neighboring image pixels) before performing compression reduces the amount of information that must be stored to achi…
PLEDGE: Progressive Lossy Error-Bounded Compression of 3D Gaussian Splatting
Oregon State University（俄勒冈州立大学）；The Ohio State University（俄亥俄州立大学）；Argonne National Laboratory（阿贡国家实验室）；The University of Texas Rio Grande Valley（德克萨斯大学里奥格兰德河谷分校）
Qualcomm AI Research, Qualcomm Technologies, Inc.（高通技术公司高通人工智能研究院）；University of California Irvine（加州大学欧文分校）；Qualcomm AI Research, Qualcomm Technologies Netherlands B.V.（高通技术荷兰有限公司高通人工智能研究院）
We present PLONQ, a progressive neural image compression scheme which pushes the boundary of variable bitrate compression by allowing quality scalable coding with a single bitstream. In contrast to existing learned variable bitrate solutions which produce separate bitstreams for each quality, it enables easier rate-control and requires less storage. Leveraging the latent scaling based variable bitrate solution, we introduce nested quantization, a method that defines multiple quantization levels with nested quantization grids, and progressively refines all latents from the coarsest to the fines…
Trit-plane coding enables deep progressive image compression, but it cannot use autoregressive context models. In this paper, we propose the context-based trit-plane coding (CTC) algorithm to achieve progressive compression more compactly. First, we develop the context-based rate reduction module to estimate trit probabilities of latent elements accurately and thus encode the trit-planes compactly. Second, we develop the context-based distortion reduction module to refine partial latent tensors from the trit-planes and improve the reconstructed image quality. Third, we propose a retraining sch…
Korea University（高丽大学）；Gauss Labs（高斯实验室）；Samsung Electronics（三星电子）
We propose the deep progressive image compression using trit-planes (DPICT) algorithm, which is the first learning-based codec supporting fine granular scalability (FGS). First, we transform an image into a latent tensor using an analysis network. Then, we represent the latent tensor in ternary digits (trits) and encode it into a compressed bitstream trit-plane by trit-plane in the decreasing order of significance. Moreover, within each trit-plane, we sort the trits according to their rate-distortion priorities and transmit more important information first. Since the compression network is les…
4DGCPro: Efficient Hierarchical 4D Gaussian Compression for Progressive Volumetric Video Streaming
Cooperative Medianet Innovation Center, Shanghai Jiaotong University（合作中位网创新中心，上海交通大学）；Department of Electronics, Shanghai Jiaotong University（电子工程系，上海交通大学）；Shanghai AI Lab（上海人工智能实验室）；Cloud platform department, E-surfing Vision Technology Co., Ltd.（云平台部门，E-surfing Vision Technology有限公司）；School of Information Science and Technology, ShanghaiTech University（信息科学与技术学院，上海科技大学）
Coarse-to-Fine: Progressive Image Compression for Semantically Hierarchical Classification
School of Integrated Technology, Yonsei University（延世大学整合技术学院）；BK21 Graduate Program in Intelligent Semiconductor Technology, Yonsei University（延世大学智能半导体技术研究生项目）；Department of Artificial Intelligence, Chung-Ang University（成均馆大学人工智能系）
Error-Propagation-Free Learned Video Compression With Dual-Domain Progressive Temporal Alignment
Department of Electronic Engineering, Shanghai Jiao Tong University（上海交通大学电子工程系）；Department of Computer Science and Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）；College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；Naval Aviation University（海军航空大学）
Nokia Technologies（诺基亚科技）；Tampere University（坦佩雷大学）
Lossless image compression is an important technique for image storage and transmission when information loss is not allowed. With the fast development of deep learning techniques, deep neural networks have been used in this field to achieve a higher compression rate. Methods based on pixel-wise autoregressive statistical models have shown good performance. However, the sequential processing way prevents these methods to be used in practice. Recently, multi-scale autoregressive models have been proposed to address this limitation. Multi-scale approaches can use parallel computing systems effic…
Progressive Learning with Visual Prompt Tuning for Variable-Rate Image Compression
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）；Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies（广东省新型安全智能技术重点实验室）；Huawei Technologies Company Ltd（华为技术有限公司）；Shenzhen University（深圳大学）
Deep learning is overwhelmingly dominant in the field of computer vision and image/video processing for the last decade. However, for image and video compression, it lags behind the traditional techniques based on discrete cosine transform (DCT) and linear filters. Built on top of an autoencoder architecture, learned image compression (LIC) systems have drawn enormous attention in recent years. Nevertheless, the proposed LIC systems are still inferior to the state-of-the-art traditional techniques, for example, the Versatile Video Coding (VVC/H.266) standard, due to either their compression pe…
This paper describes a channel-selection approach for simplifying deep neural networks. Specifically, we propose a new type of generic network layer, called pruning layer, to seamlessly augment a given pre-trained model for compression. Each pruning layer, comprising $1 \times 1$ depth-wise kernels, is represented with a dual format: one is real-valued and the other is binary. The former enables a two-phase optimization process of network pruning to operate with an end-to-end differentiable network, and the latter yields the mask information for channel selection. Our method progressively perf…
Singapore University of Technology and Design（新加坡科技设计大学）；Nanyang Technological University（南洋理工大学）；Monash University（莫纳什大学）；New York University（纽约大学）；Lancaster University（兰卡斯特大学）
Sensetime Research（商汤科技研究院）；Tsinghua University（清华大学）；Syracuse University（雪城大学）；Northeastern University（东北大学）；Florida International University（佛罗里达国际大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research（IBM研究院）
Weight pruning and weight quantization are two important categories of DNN model compression. Prior work on these techniques are mainly based on heuristics. A recent work developed a systematic frame-work of DNN weight pruning using the advanced optimization technique ADMM (Alternating Direction Methods of Multipliers), achieving one of state-of-art in weight pruning results. In this work, we first extend such one-shot ADMM-based framework to guarantee solution feasibility and provide fast convergence rate, and generalize to weight quantization as well. We have further developed a multi-step, …
SPAC: Sampling-based Progressive Attribute Compression for Dense Point Clouds
School of Control Science and Engineering, Shandong University（山东大学控制科学与工程学院）；School of Engineering and Sustainable Development, De Montfort University（德蒙福特大学工程与可持续发展学院）；School of Data Science, Lingnan University（岭南大学数据科学学院）
Rochester Institute of Technology（罗切斯特理工学院）；Pennsylvania State University（宾夕法尼亚州立大学）；Old Dominion University（奥尔德多米宁大学）
For lossy image compression systems, we develop an algorithm, iterative refinement, to improve the decoder's reconstruction compared to standard decoding techniques. Specifically, we propose a recurrent neural network approach for nonlinear, iterative decoding. Our decoder, which works with any encoder, employs self-connected memory units that make use of causal and non-causal spatial context information to progressively reduce reconstruction error over a fixed number of steps. We experiment with variants of our estimator and find that iterative refinement consistently creates lower distortion…
Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression
School of Information Science and Technology, Fudan University（复旦大学信息科学与技术学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Academy for Engineering and Technology, Fudan University（复旦大学工程技术学院）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）
Progressively Deformable 2D Gaussian Splatting for Video Representation at Arbitrary Resolutions
Shanghai Jiao Tong University, Shanghai, China（上海交通大学）；University of Missouri--Kansas City, Kansas City, MO, USA（密苏里大学-堪萨斯城分校）；University of Canterbury, Christchurch, New Zealand（坎特伯雷大学）
Hierarchical Semantic Compression for Consistent Image Semantic Restoration
School of Electronic and Information Engineering, Beihang University（北京航空航天大学电子信息工程学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Electronic and Information Engineering, University of Electronic Science and Technology of China（电子科技大学电子信息工程学院）
Computational photography involves sophisticated capture methods. A new trend is to capture projection of higher dimensional visual signals such as videos, multi-spectral data and lightfields on lower dimensional sensors. Carefully designed capture methods exploit the sparsity of the underlying signal in a transformed domain to reduce the number of measurements and use an appropriate reconstruction method. Traditional progressive methods may capture successively more detail using a sequence of simple projection basis, such as DCT or wavelets and employ straightforward backprojection for recons…
