Group-Wise Optimization for Self-Extensible Codebooks in Vector Quantized Models
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, China（人工智能学院，南京航空航天大学）；MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China（国家工业和信息化部模式分析与机器智能重点实验室，南京）；The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China（教育部脑机智能技术重点实验室，南京）
Vector Quantized Variational Autoencoders (VQ-VAEs) leverage self-supervised learning through reconstruction tasks to represent continuous vectors using the closest vectors in a codebook. However, issues such as codebook collapse persist in the VQ model. To address these issues, existing approaches employ implicit static codebooks or jointly optimize the entire codebook, but these methods constrain the codebook's learning capability, leading to reduced reconstruction quality. In this paper, we propose Group-VQ, which performs group-wise optimization on the codebook. Each group is optimized independently, with joint optimization performed within groups. This approach improves the trade-off between codebook utilization and reconstruction performance. Additionally, we introduce a training-free codebook resampling method, allowing post-training adjustment of the codebook size. In image reconstruction experiments under various settings, Group-VQ demonstrates improved performance on reconstruction metrics. And the post-training codebook sampling method achieves the desired flexibility in adjusting the codebook size.
All-in-One Medical Image Restoration with Latent Diffusion-Enhanced Vector-Quantized Codebook Prior
School of Biological Science and Medical Engineering（生物科学与医学工程学院）；State Key Laboratory of Software Development Environment（软件开发环境国家重点实验室）；Key Laboratory of Biomechanics and Mechanobiology of Ministry of Education（教育部生物力学与机械生物学重点实验室）；Beijing Advanced Innovation Center for Biomedical Engineering（北京生物医学工程先进创新中心）；Beihang University（北京航空航天大学）；Department of Biomedical Engineering（生物医学工程系）；Tsinghua University（清华大学）；ByteDance Inc.（字节跳动公司）
VQCNIR: Clearer Night Image Restoration with Vector-Quantized Codebook
South China University of Technology（华南理工大学）；Pazhou Laboratory（琶洲实验室）；The Hong Kong University of Science and Technology（香港科技大学）；Fujian Normal University（福建师范大学）
Nankai University（南开大学）；Tencent PCG（腾讯平台与内容事业群）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanghai AI Laboratory（上海人工智能实验室）；Tencent Online Video（腾讯在线视频）
Although generative facial prior and geometric prior have recently demonstrated high-quality results for blind face restoration, producing fine-grained facial details faithful to inputs remains a challenging problem. Motivated by the classical dictionary-based methods and the recent vector quantization (VQ) technique, we propose a VQ-based face restoration method - VQFR. VQFR takes advantage of high-quality low-level feature banks extracted from high-quality faces and can thus help recover realistic facial details. However, the simple application of the VQ codebook cannot achieve good results …
Vector-Quantized Variational Autoencoders (VQ-VAE)[1] provide an unsupervised model for learning discrete representations by combining vector quantization and autoencoders. In this paper, we study the use of VQ-VAE for representation learning for downstream tasks, such as image retrieval. We first describe the VQ-VAE in the context of an information-theoretic framework. We show that the regularization term on the learned representation is determined by the size of the embedded codebook before the training and it affects the generalization ability of the model. As a result, we introduce a hyper…
Concept-Centric Token Interpretation for Vector-Quantized Generative Models
School of Computing, University of Georgia（佐治亚大学计算学院）；Department of Data Science, New Jersey Institute of Technology（新泽西理工学院数据科学系）；Department of Computer Science, New York University（纽约大学计算机科学系）
QUVA Lab, University of Amsterdam（QUVA实验室，阿姆斯特丹大学）；AMLab, Informatics Institute, University of Amsterdam（AMLab，阿姆斯特丹大学信息学院）；AI4Science Lab, University of Amsterdam（AI4Science实验室，阿姆斯特丹大学）；Korteweg-de Vries Institute for Mathematics, University of Amsterdam（科特韦格 - 德弗里斯数学研究所，阿姆斯特丹大学）；Qualcomm AI Research（高通人工智能研究公司）；FunAI Lab, University of Technology Nuremberg（FunAI实验室，纽伦堡工业大学）
Monash University（莫纳什大学）；KTH Royal Institute of Technology（KTH皇家理工学院）
We propose a multi-layer variational autoencoder method, we call HR-VQVAE, that learns hierarchical discrete representations of the data. By utilizing a novel objective function, each layer in HR-VQVAE learns a discrete representation of the residual from previous layers through a vector quantized encoder. Furthermore, the representations at each layer are hierarchically linked to those at previous layers. We evaluate our method on the tasks of image reconstruction and generation. Experimental results demonstrate that the discrete representations learned by HR-VQVAE enable the decoder to recon…
Although two-stage Vector Quantized (VQ) generative models allow for synthesizing high-fidelity and high-resolution images, their quantization operator encodes similar patches within an image into the same index, resulting in a repeated artifact for similar adjacent regions using existing decoder architectures. To address this issue, we propose to incorporate the spatially conditional normalization to modulate the quantized vectors so as to insert spatially variant information to the embedded index maps, encouraging the decoder to generate more photorealistic images. Moreover, we use multichan…
Addressing Representation Collapse in Vector Quantized Models with One Linear Layer
University of Science and Technology of China（中国科学技术大学）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）；Peking University（北京大学）；Jinan University（济南大学）
Sony Group Corporation（索尼集团公司）；Sony Corporation of America（美国索尼公司）
One noted issue of vector-quantized variational autoencoder (VQ-VAE) is that the learned discrete representation uses only a fraction of the full capacity of the codebook, also known as codebook collapse. We hypothesize that the training scheme of VQ-VAE, which involves some carefully designed heuristics, underlies this issue. In this paper, we propose a new training scheme that extends the standard VAE via novel stochastic dequantization and quantization, called stochastically quantized variational autoencoder (SQ-VAE). In SQ-VAE, we observe a trend that the quantization is stochastic at the …
National University of Singapore（新加坡国立大学）；Tencent PCG（腾讯平台与内容事业群）
Vector-Quantized (VQ-based) generative models usually consist of two basic components, i.e., VQ tokenizers and generative transformers. Prior research focuses on improving the reconstruction fidelity of VQ tokenizers but rarely examines how the improvement in reconstruction affects the generation ability of generative transformers. In this paper, we surprisingly find that improving the reconstruction fidelity of VQ tokenizers does not necessarily improve the generation. Instead, learning to compress semantic features within VQ tokenizers significantly improves generative transformers' ability …
VQ-DeepISC: Vector Quantized-Enabled Digital Semantic Communication with Channel Adaptive Image Transmission
ZGC Institute of Ubiquitous-X Innovation and Applications（ubiquitous-X创新与应用研究院）；State Key Laboratory of Networking and Switching Technology（网络与交换技术国家重点实验室）；Beijing University of Posts and Telecommunications（北京邮电大学）
Pretraining language models with next-token prediction on massive text corpora has delivered phenomenal zero-shot, few-shot, transfer learning and multi-tasking capabilities on both generative and discriminative language tasks. Motivated by this success, we explore a Vector-quantized Image Modeling (VIM) approach that involves pretraining a Transformer to predict rasterized image tokens autoregressively. The discrete image tokens are encoded from a learned Vision-Transformer-based VQGAN (ViT-VQGAN). We first propose multiple improvements over vanilla VQGAN from architecture to codebook learnin…
ArcVQ-VAE: A Spherical Vector Quantization Framework with ArcCosine Additive Margin
Department of Artificial Intelligence, Chung-Ang University, Seoul, Republic of Korea（韩国首尔 Chung-Ang 大学人工智能系）；SNUAILAB, Seoul, Republic of Korea（韩国首尔 SNUAILAB 实验室）
