This paper challenges the dominance of continuous pipelines in visual generation. We systematically investigate the performance gap between discrete and continuous methods. Contrary to the belief that discrete tokenizers are intrinsically inferior, we demonstrate that the disparity arises primarily from the total number of bits allocated in the latent space (i.e., the compression ratio). We show that scaling up the codebook size effectively bridges this gap, allowing discrete tokenizers to match or surpass their continuous counterparts. However, existing discrete generation methods struggle to capitalize on this insight, suffering from performance degradation or prohibitive training costs with scaled codebook. To address this, we propose masked Bit AutoRegressive modeling (BAR), a scalable framework that supports arbitrary codebook sizes. By equipping an autoregressive transformer with a masked bit modeling head, BAR predicts discrete tokens through progressively generating their constituent bits. BAR achieves a new state-of-the-art gFID of 0.99 on ImageNet-256, outperforming leading methods across both continuous and discrete paradigms, while significantly reducing sampling costs and converging faster than prior continuous approaches. Project page is available at https://bar-gen.github.io/
Self-control: A Better Conditional Mechanism for Masked Autoregressive Model
Nankai University（南开大学）；School of Statistics and Data Science（统计与数据科学学院）；Key Laboratory of Pure Mathematics and Combinatorics, Ministry of Education（教育部纯数学与组合数学重点实验室）
Non-autoregressive generative transformers recently demonstrated impressive image generation performance, and orders of magnitude faster sampling than their autoregressive counterparts. However, optimal parallel sampling from the true joint distribution of visual tokens remains an open challenge. In this paper we introduce Token-Critic, an auxiliary model to guide the sampling of a non-autoregressive generative transformer. Given a masked-and-reconstructed real image, the Token-Critic model is trained to distinguish which visual tokens belong to the original image and which were sampled by the…
Not All Image Regions Matter: Masked Vector Quantization for Autoregressive Image Generation
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）
Conditional Panoramic Image Generation via Masked Autoregressive Modeling
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；Insta360 Research（Insta360研究院）；National University of Singapore（新加坡国立大学）；The University of Tokyo（东京大学）
LazyMAR: Accelerating Masked Autoregressive Models via Feature Caching
Beijing Institute of Technology（北京理工大学）；Central South University（中南大学）；China University of Mining and Technology（中国矿业大学）；University of Electronic Science and Technology of China（电子科技大学）；Tsinghua University（清华大学）；Shanghai Jiaotong University（上海交通大学）
针对 MAR 双向自注意力无法使用传统 KV 缓存的问题，本文提出训练免费、即插即用的 LazyMAR，通过令牌与条件两类特征缓存实现 2.83 倍加速且几乎不损失生成质量。
CanvasMAR: Improving Masked Autoregressive Video Prediction With Canvas
Institute for Artificial Intelligence, Peking University, Beijing, China（人工智能研究院，北京大学，北京，中国）；School of Intelligence Science and Technology, Peking University, Beijing, China（智能科学与技术学院，北京大学，北京，中国）；State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China（通用人工智能国家重点实验室，北京大学，北京，中国）
DiverseAR: Boosting Diversity in Bitwise Autoregressive Image Generation
PRLab, Nanjing University（南京大学PRLab）；The University of Hong Kong（香港大学）；University of Chinese Academy of Sciences（中国科学院大学）；Lovart AI；WeChat, Tencent Inc.（腾讯公司）
Image inpainting is an underdetermined inverse problem, which naturally allows diverse contents to fill up the missing or corrupted regions realistically. Prevalent approaches using convolutional neural networks (CNNs) can synthesize visually pleasant contents, but CNNs suffer from limited perception fields for capturing global features. With image-level attention, transformers enable to model long-range dependencies and generate diverse contents with autoregressive modeling of pixel-sequence distributions. However, the unidirectional attention in autoregressive transformers is suboptimal as c…
Generative transformers have experienced rapid popularity growth in the computer vision community in synthesizing high-fidelity and high-resolution images. The best generative transformer models so far, however, still treat an image naively as a sequence of tokens, and decode an image sequentially following the raster scan ordering (i.e. line-by-line). We find this strategy neither optimal nor efficient. This paper proposes a novel image synthesis paradigm using a bidirectional transformer decoder, which we term MaskGIT. During training, MaskGIT learns to predict randomly masked tokens by atte…
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Nanjing University（南京大学）；Peking University（北京大学）；Peng Cheng Laboratory（鹏城实验室）；City University of Hong Kong（香港城市大学）
Existing captioning models often adopt the encoder-decoder architecture, where the decoder uses autoregressive decoding to generate captions, such that each token is generated sequentially given the preceding generated tokens. However, autoregressive decoding results in issues such as sequential error accumulation, slow generation, improper semantics and lack of diversity. Non-autoregressive decoding has been proposed to tackle slow generation for neural machine translation but suffers from multimodality problem due to the indirect modeling of the target distribution. In this paper, we propose…
We present Muse, a text-to-image Transformer model that achieves state-of-the-art image generation performance while being significantly more efficient than diffusion or autoregressive models. Muse is trained on a masked modeling task in discrete token space: given the text embedding extracted from a pre-trained large language model (LLM), Muse is trained to predict randomly masked image tokens. Compared to pixel-space diffusion models, such as Imagen and DALL-E 2, Muse is significantly more efficient due to the use of discrete tokens and requiring fewer sampling iterations; compared to autore…
While generative modeling on multimodal image-text data has been actively developed with large-scale paired datasets, there have been limited attempts to generate both image and text data by a single model rather than a generation of one fixed modality conditioned on the other modality. In this paper, we explore a unified generative vision-and-language (VL) model that can produce both images and text sequences. Especially, we propose a generative VL transformer based on the non-autoregressive mask prediction, named MAGVLT, and compare it with an autoregressive generative VL transformer (ARGVLT…
We present Bit Diffusion: a simple and generic approach for generating discrete data with continuous state and continuous time diffusion models. The main idea behind our approach is to first represent the discrete data as binary bits, and then train a continuous diffusion model to model these bits as real numbers which we call analog bits. To generate samples, the model first generates the analog bits, which are then thresholded to obtain the bits that represent the discrete variables. We further propose two simple techniques, namely Self-Conditioning and Asymmetric Time Intervals, which lead …
In this work, we propose a new unsupervised image segmentation approach based on mutual information maximization between different constructed views of the inputs. Taking inspiration from autoregressive generative models that predict the current pixel from past pixels in a raster-scan ordering created with masked convolutions, we propose to use different orderings over the inputs using various forms of masked convolutions to construct different views of the data. For a given input, the model produces a pair of predictions with two valid orderings, and is then trained to maximize the mutual inf…
Masked AutoDecoder is Effective Multi-Task Vision Generalist
S-Lab, Nanyang Technological University（南洋理工大学S-Lab）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Sensetime Research（商汤科技研究院）；College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）
High-dimensional generative models have many applications including image compression, multimedia generation, anomaly detection and data completion. State-of-the-art estimators for natural images are autoregressive, decomposing the joint distribution over pixels into a product of conditionals parameterized by a deep neural network, e.g. a convolutional neural network such as the PixelCNN. However, PixelCNNs only model a single decomposition of the joint, and only a single generation order is efficient. For tasks such as image completion, these models are unable to use much of the observed cont…
Flow-based generative models, conceptually attractive due to tractability of both the exact log-likelihood computation and latent-variable inference, and efficiency of both training and sampling, has led to a number of impressive empirical successes and spawned many advanced variants and theoretical investigations. Despite their computational efficiency, the density estimation performance of flow-based generative models significantly falls behind those of state-of-the-art autoregressive models. In this work, we introduce masked convolutional generative flow (MaCow), a simple yet effective arch…
FastMap: Real-Time Semantic Map Completion via Bitwise Masked Modeling
NYUAD Center for Artificial Intelligence and Robotics (CAIR)（纽约大学阿布扎克人工智能与机器人中心）；New York University Abu Dhabi（纽约大学阿布扎克分校）；New York University（纽约大学）；Embodied AI and Robotics (AIR) Lab（具身人工智能与机器人实验室）；School of Software, Tsinghua University（清华大学软件学院）
Deep autoregressive models are one of the most powerful models that exist today which achieve state-of-the-art bits per dim. However, they lie at a strict disadvantage when it comes to controlled sample generation compared to latent variable models. Latent variable models such as VAEs and normalizing flows allow meaningful semantic manipulations in latent space, which autoregressive models do not have. In this paper, we propose using Fisher scores as a method to extract embeddings from an autoregressive model to use for interpolation and show that our method provides more meaningful sample man…
Recent models for learned image compression are based on autoencoders, learning approximately invertible mappings from pixels to a quantized latent representation. These are combined with an entropy model, a prior on the latent representation that can be used with standard arithmetic coding algorithms to yield a compressed bitstream. Recently, hierarchical entropy models have been introduced as a way to exploit more structure in the latents than simple fully factorized priors, improving compression performance while maintaining end-to-end optimization. Inspired by the success of autoregressive…
Group-of-Latents: Perceptual Video Compression at Extreme Bitrates via Masked Latent Generative Modeling
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China（多媒体信息处理国家重点实验室，计算机科学学院，北京大学，北京，中国）；State Key Discipline Laboratory of Wide Band-Gap Semiconductor Technology, School of Microelectronics, Xidian University, Xi'an, China（宽带隙半导体技术重点学科实验室，微电子学院，西安电子科技大学，西安，中国）
