Conceptual Compression via Deep Structure and Texture Synthesis
Peking University（北京大学）；Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Peng Cheng Laboratory（鹏城实验室）；City University of Hong Kong（香港城市大学）；Communication University of China（中国传媒大学）
Existing compression methods typically focus on the removal of signal-level redundancies, while the potential and versatility of decomposing visual data into compact conceptual components still lack further study. To this end, we propose a novel conceptual compression framework that encodes visual data into compact structure and texture representations, then decodes in a deep synthesis fashion, aiming to achieve better visual reconstruction quality, flexible content manipulation, and potential support for various vision tasks. In particular, we propose to compress images by a dual-layered model consisting of two complementary visual features: 1) structure layer represented by structural maps and 2) texture layer characterized by low-dimensional deep representations. At the encoder side, the structural maps and texture representations are individually extracted and compressed, generating the compact, interpretable, inter-operable bitstreams. During the decoding stage, a hierarchical fusion GAN (HF-GAN) is proposed to learn the synthesis paradigm where the textures are rendered into the decoded structural maps, leading to high-quality reconstruction with remarkable visual realism. Extensive experiments on diverse images have demonstrated the superiority of our framework with lower bitrates, higher reconstruction quality, and increased versatility towards visual analysis and content manipulation tasks.
The end-to-end learning-based video compression has attracted substantial attentions by paving another way to compress video signals as stacked visual features. This paper proposes an efficient end-to-end deep video codec with jointly optimized compression and enhancement modules (JCEVC). First, we propose a dual-path generative adversarial network (DPEG) to reconstruct video details after compression. An $α$-path facilitates the structure information reconstruction with a large receptive field and multi-frame references, while a $β$-path facilitates the reconstruction of local textures. Both …
Non-Stationary Texture Synthesis by Adversarial Expansion
Shenzhen University（深圳大学）；Huazhong University of Science & Technology（华中科技大学）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）；Tel Aviv University（特拉维夫大学）
University of Oxford（牛津大学）；The University of Hong Kong（香港大学）
Generative Adversarial Networks (GANs) can generate realistic fake face images that can easily fool human beings.On the contrary, a common Convolutional Neural Network(CNN) discriminator can achieve more than 99.9% accuracyin discerning fake/real images. In this paper, we conduct an empirical study on fake/real faces, and have two important observations: firstly, the texture of fake faces is substantially different from real ones; secondly, global texture statistics are more robust to image editing and transferable to fake faces from different GANs and datasets. Motivated by the above observat…
City University of Hong Kong（香港城市大学）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）
It plays a fundamental role to compactly represent the visual information towards the optimization of the ultimate utility in myriad visual data centered applications. With numerous approaches proposed to efficiently compress the texture and visual features serving human visual perception and machine intelligence respectively, much less work has been dedicated to studying the interactions between them. Here we investigate the integration of feature and texture compression, and show that a universal and collaborative visual information representation can be achieved in a hierarchical way. In pa…
Conceptual coding has been an emerging research topic recently, which encodes natural images into disentangled conceptual representations for compression. However, the compression performance of the existing methods is still sub-optimal due to the lack of comprehensive consideration of rate constraint and reconstruction quality. To this end, we propose a novel end-to-end semantic prior modeling-based conceptual coding scheme towards extremely low bitrate image compression, which leverages semantic-wise deep representations as a unified prior for entropy estimation and texture synthesis. Specif…
There has been a growing interest in using different approaches to improve the coding efficiency of modern video codec in recent years as demand for web-based video consumption increases. In this paper, we propose a model-based approach that uses texture analysis/synthesis to reconstruct blocks in texture regions of a video to achieve potential coding gains using the AV1 codec developed by the Alliance for Open Media (AOM). The proposed method uses convolutional neural networks to extract texture regions in a frame, which are then reconstructed using a global motion model. Our preliminary resu…
We introduce a simple recurrent variational auto-encoder architecture that significantly improves image modeling. The system represents the state-of-the-art in latent variable models for both the ImageNet and Omniglot datasets. We show that it naturally separates global conceptual information from lower level details, thus addressing one of the fundamentally desired properties of unsupervised learning. Furthermore, the possibility of restricting ourselves to storing only global information about an image allows us to achieve high quality 'conceptual compression'.
Arizona State University（亚利桑那州立大学）；Lebanese American University（黎巴嫩美国大学）
Learning-based image compression was shown to achieve a competitive performance with state-of-the-art transform-based codecs. This motivated the development of new learning-based visual compression standards such as JPEG-AI. Of particular interest to these emerging standards is the development of learning-based image compression systems targeting both humans and machines. This paper is concerned with learning-based compression schemes whose compressed-domain representations can be utilized to perform visual processing and computer vision tasks directly in the compressed domain. Such a characte…
Supervised pixel-based texture classification is usually performed in the feature space. We propose to perform this task in (dis)similarity space by introducing a new compression-based (dis)similarity measure. The proposed measure utilizes two dimensional MPEG-1 encoder, which takes into consideration the spatial locality and connectivity of pixels in the images. The proposed formulation has been carefully designed based on MPEG encoder functionality. To this end, by design, it solely uses P-frame coding to find the (dis)similarity among patches/images. We show that the proposed measure works …
Texture exists in lots of the products, such as wood, beef and compression tea. These abundant and stochastic texture patterns are significantly different between any two products. Unlike the traditional digital ID tracking, in this paper, we propose a novel approach for product traceability, which directly uses the natural texture of the product itself as the unique identifier. A texture identification based traceability system for Pu'er compression tea is developed to demonstrate the feasibility of the proposed solution. With tea-brick images collected from manufactures and individual users,…
We describe a novel approach for compressing truncated signed distance fields (TSDF) stored in 3D voxel grids, and their corresponding textures. To compress the TSDF, our method relies on a block-based neural network architecture trained end-to-end, achieving state-of-the-art rate-distortion trade-off. To prevent topological errors, we losslessly compress the signs of the TSDF, which also upper bounds the reconstruction error by the voxel size. To compress the corresponding texture, we designed a fast block-based UV parameterization, generating coherent texture maps that can be effectively com…
Coordinated Science Lab, University of Illinois（伊利诺伊大学协调科学实验室）；HRL Laboratories, LLC（HRL 有限责任公司实验室）；University of California, Berkeley（加州大学伯克利分校）；Microsoft Research Asia（微软亚洲研究院）
We present a novel algorithm for segmentation of natural images that harnesses the principle of minimum description length (MDL). Our method is based on observations that a homogeneously textured region of a natural image can be well modeled by a Gaussian distribution and the region boundary can be effectively coded by an adaptive chain code. The optimal segmentation of an image is the one that gives the shortest coding length for encoding all textures and boundaries in the image, and is obtained via an agglomerative clustering process applied to a hierarchy of decreasing window sizes as multi…
School of Information and Communication Engineering, University of Electronic Science and Technology of China（电子科技大学信息与通信工程学院）；Peng Cheng Laboratory（鹏城实验室）
Peking University（北京大学）；City University of Hong Kong（香港城市大学）
In this paper, we propose a scalable image compression scheme, including the base layer for feature representation and enhancement layer for texture representation. More specifically, the base layer is designed as the deep learning feature for analysis purpose, and it can also be converted to the fine structure with deep feature reconstruction. The enhancement layer, which serves to compress the residuals between the input image and the signals generated from the base layer, aims to faithfully reconstruct the input texture. The proposed scheme can feasibly inherit the advantages of both compre…
Freie Universität（柏林自由大学）；University of Chicago（芝加哥大学）；University of Malta（马耳他大学）；West Virginia University（西弗吉尼亚大学）；Centro de Astrobiología, CSIC-INTA（西班牙天体生物学中心）
(abridged) We describe an image-comparison technique of Heidemann and Ritter that uses image compression, and is capable of: (i) detecting novel textures in a series of images, as well as of: (ii) alerting the user to the similarity of a new image to a previously-observed texture. This image-comparison technique has been implemented and tested using our Astrobiology Phone-cam system, which employs Bluetooth communication to send images to a local laptop server in the field for the image-compression analysis. We tested the system in a field site displaying a heterogeneous suite of sandstones, l…
In the past years, numerous methods have been introduced to reliably detect digital face image manipulations. Lately, the generalizability of these schemes has been questioned in particular with respect to image post-processing. Image compression represents a post-processing which is frequently applied in diverse biometric application scenarios. Severe compression might erase digital traces of face image manipulation and hence hamper a reliable detection thereof. In this work, the effects of image compression on face image manipulation detection are analyzed. In particular, a case study on fac…
Univ Lyon, CNRS, LIRIS（里昂大学，法国国家科学研究中心，LIRIS）；Multimedia Research Center, University of Alberta（阿尔伯塔大学多媒体研究中心）
Objective visual quality assessment of 3D models is a fundamental issue in computer graphics. Quality assessment metrics may allow a wide range of processes to be guided and evaluated, such as level of detail creation, compression, filtering, and so on. Most computer graphics assets are composed of geometric surfaces on which several texture images can be mapped to 11 make the rendering more realistic. While some quality assessment metrics exist for geometric surfaces, almost no research has been conducted on the evaluation of texture-mapped 3D models. In this context, we present a new subject…
New Horizon College of Engineering（新视野工程学院）；Government Engineering College（政府工程学院）；University of Mysore（迈索尔大学）
Periodicity of a texture is one of the important visual characteristics and is often used as a measure for textural discrimination at the structural level. Knowledge about periodicity of a texture is very essential in the field of texture synthesis and texture compression and also in the design of frieze and wall papers. In this paper, we propose a method of periodicity extraction from noisy images based on superposition of distance matching function (DMF) and wavelet decomposition without de-noising the test images. Overall DMFs are subjected to single-level Haar wavelet decomposition to obta…
Image defocus is inherent in the physics of image formation caused by the optical aberration of lenses, providing plentiful information on image quality. Unfortunately, existing quality enhancement approaches for compressed images neglect the inherent characteristic of defocus, resulting in inferior performance. This paper finds that in compressed images, significantly defocused regions have better compression quality, and two regions with different defocus values possess diverse texture patterns. These observations motivate our defocus-aware quality enhancement (DAQE) approach. Specifically, …
Conceptual Codebook Learning for Vision-Language Models
Harbin Institute of Technology（哈尔滨工业大学）；Southern University of Science and Technology（南方科技大学）；University of California San Diego（加利福尼亚大学圣迭戈分校）；University of Missouri（密苏里大学）
Due to a resource-constrained environment, network compression has become an important part of deep neural networks research. In this paper, we propose a new compression method, \textit{Inter-Layer Weight Prediction} (ILWP) and quantization method which quantize the predicted residuals between the weights in all convolution layers based on an inter-frame prediction method in conventional video coding schemes. Furthermore, we found a phenomenon \textit{Smoothly Varying Weight Hypothesis} (SVWH) which is that the weights in adjacent convolution layers share strong similarity in shapes and values…
When approaching a novel visual recognition problem in a specialized image domain, a common strategy is to start with a pre-trained deep neural network and fine-tune it to the specialized domain. If the target domain covers a smaller visual space than the source domain used for pre-training (e.g. ImageNet), the fine-tuned network is likely to be over-parameterized. However, applying network pruning as a post-processing step to reduce the memory requirements has drawbacks: fine-tuning and pruning are performed independently; pruning parameters are set once and cannot adapt over time; and the hi…
Stanford University（斯坦福大学）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
This paper investigates deep neural network (DNN) compression from the perspective of compactly representing and storing trained parameters. We explore the previously overlooked opportunity of cross-layer architecture-agnostic representation sharing for DNN parameters. To do this, we decouple feedforward parameters from DNN architectures and leverage additive quantization, an extreme lossy compression method invented for image descriptors, to compactly represent the parameters. The representations are then finetuned on task objectives to improve task accuracy. We conduct extensive experiments …
Group-of-Latents: Perceptual Video Compression at Extreme Bitrates via Masked Latent Generative Modeling
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China（多媒体信息处理国家重点实验室，计算机科学学院，北京大学，北京，中国）；State Key Discipline Laboratory of Wide Band-Gap Semiconductor Technology, School of Microelectronics, Xidian University, Xi'an, China（宽带隙半导体技术重点学科实验室，微电子学院，西安电子科技大学，西安，中国）
Recent years have witnessed the rapid development of image storage and transmission systems, in which image compression plays an important role. Generally speaking, image compression algorithms are developed to ensure good visual quality at limited bit rates. However, due to the different compression optimization methods, the compressed images may have different levels of quality, which needs to be evaluated quantificationally. Nowadays, the mainstream full-reference (FR) metrics are effective to predict the quality of compressed images at coarse-grained levels (the bit rates differences of co…
