In generative modeling, tokenization simplifies complex data into compact, structured representations, creating a more efficient, learnable space. For high-dimensional visual data, it reduces redundancy and emphasizes key features for high-quality generation. Current visual tokenization methods rely on a traditional autoencoder framework, where the encoder compresses data into latent representations, and the decoder reconstructs the original input. In this work, we offer a new perspective by proposing denoising as decoding, shifting from single-step reconstruction to iterative refinement. Specifically, we replace the decoder with a diffusion process that iteratively refines noise to recover the original image, guided by the latents provided by the encoder. We evaluate our approach by assessing both reconstruction (rFID) and generation quality (FID), comparing it to state-of-the-art autoencoding approaches. By adopting iterative reconstruction through diffusion, our autoencoder, namely Epsilon-VAE, achieves high reconstruction quality, which in turn enhances downstream generation quality by 22% at the same compression rates or provides 2.3x inference speedup through increasing compression rates. We hope this work offers new insights into integrating iterative generation and autoencoding for improved compression and generation.
D-CNN and VQ-VAE Autoencoders for Compression and Denoising of Industrial X-ray Computed Tomography Images
Department for X-Ray Imaging, Federal Institute for Materials Research（X射线成像部门，材料研究与测试联邦机构）；Institute of Physics and Astronomy, University of Potsdam（物理与天文学研究所，波茨坦大学）
Denoising Vision Transformer Autoencoder with Spectral Self-Regularization
Nanjing University（南京大学）；Kling Team, Kuaishou Technology（快手科技 Kling 团队）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shanghai Jiao Tong University（上海交通大学）；University of Science and Technology of China（中国科学技术大学）
Peking University（北京大学）；Microsoft Research Asia（微软亚洲研究院）
Recently most successful image synthesis models are multi stage process to combine the advantages of different methods, which always includes a VAE-like model for faithfully reconstructing embedding to image and a prior model to generate image embedding. At the same time, diffusion models have shown be capacity to generate high-quality synthetic images. Our work proposes a VQ-VAE architecture model with a diffusion decoder (DiVAE) to work as the reconstructing component in image synthesis. We explore how to input image embedding into diffusion model for excellent performance and find that simp…
Conditional generative models of high-dimensional images have many applications, but supervision signals from conditions to images can be expensive to acquire. This paper describes Diffusion-Decoding models with Contrastive representations (D2C), a paradigm for training unconditional variational autoencoders (VAEs) for few-shot conditional image generation. D2C uses a learned diffusion-based prior over the latent representations to improve generation and contrastive self-supervised learning to improve representation quality. D2C can adapt to novel generation tasks conditioned on labels or mani…
Image synthesis under multi-modal priors is a useful and challenging task that has received increasing attention in recent years. A major challenge in using generative models to accomplish this task is the lack of paired data containing all modalities (i.e. priors) and corresponding outputs. In recent work, a variational auto-encoder (VAE) model was trained in a weakly supervised manner to address this challenge. Since the generative power of VAEs is usually limited, it is difficult for this method to synthesize images belonging to complex distributions. To this end, we propose a solution base…
Disentangled Hierarchical VAE for 3D Human-Human Interaction Generation
Department of CSSE, The University of Western Australia（西澳大学计算机科学与工程系）；Data61, CSIRO（澳大利亚联邦科学工业研究组织Data61部门）；Australian Institute for Machine Learning, The University of Adelaide（澳大利亚阿德莱德大学人工智能研究所）；NERC-RVC, Hunan University（湖南大学NERC-RVC部门）
SeNM-VAE: Semi-Supervised Noise Modeling with Hierarchical Variational Autoencoder
Yau Mathematical Sciences Center, Tsinghua University（清华大学丘成桐数学科学中心）；Hisilicon（海思半导体）；Yanqi Lake Beijing Institute of Mathematical Sciences and Applications（雁栖湖北京数学科学与应用研究院）；State Key Laboratory of Membrane Biology, School of Life Sciences, Tsinghua University（清华大学生命科学学院膜生物学国家重点实验室）
Self-supervised denoising of visual field data improves detection of glaucoma progression
Pepperdine University（佩珀代因大学）；University of California Los Angeles（加利福尼亚大学洛杉矶分校）；Stein Eye Institute（斯坦眼科研究所）；David Geffen School of Medicine（戴维·格芬医学院）；Columbia University（哥伦比亚大学）；University of California Riverside（加利福尼亚大学河滨分校）
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）；Microsoft Cloud+AI（微软云与人工智能部门）
We present the vector quantized diffusion (VQ-Diffusion) model for text-to-image generation. This method is based on a vector quantized variational autoencoder (VQ-VAE) whose latent space is modeled by a conditional variant of the recently developed Denoising Diffusion Probabilistic Model (DDPM). We find that this latent-space method is well-suited for text-to-image generation tasks because it not only eliminates the unidirectional bias with existing methods but also allows us to incorporate a mask-and-replace diffusion strategy to avoid the accumulation of errors, which is a serious problem w…
