The Open-MAGVIT2 project produces an open-source replication of Google's MAGVIT-v2 tokenizer, a tokenizer with a super-large codebook (i.e., $2^{18}$ codes), and achieves the state-of-the-art reconstruction performance on ImageNet and UCF benchmarks. We also provide a tokenizer pre-trained on large-scale data, significantly outperforming Cosmos on zero-shot benchmarks (1.93 vs. 0.78 rFID on ImageNet original resolution). Furthermore, we explore its application in plain auto-regressive models to validate scalability properties, producing a family of auto-regressive image generation models ranging from 300M to 1.5B. To assist auto-regressive models in predicting with a super-large vocabulary, we factorize it into two sub-vocabulary of different sizes by asymmetric token factorization, and further introduce ``next sub-token prediction'' to enhance sub-token interaction for better generation quality. We release all models and codes to foster innovation and creativity in the field of auto-regressive visual generation.
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding
The University of Hong Kong（香港大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；CUHK（香港中文大学）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Infinigence AI（智源人工智能）
University of Waterloo（滑铁卢大学）；Alibaba Group（阿里巴巴集团）；Vector Institute（向量研究所）
Conditioned diffusion models have demonstrated state-of-the-art text-to-image synthesis capacity. Recently, most works focus on synthesizing independent images; While for real-world applications, it is common and necessary to generate a series of coherent images for story-stelling. In this work, we mainly focus on story visualization and continuation tasks and propose AR-LDM, a latent diffusion model auto-regressively conditioned on history captions and generated images. Moreover, AR-LDM can generalize to new characters through adaptation. To our best knowledge, this is the first work successf…
Nanyang Technological University（南洋理工大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Amazon（亚马逊）
Deep generative models have achieved conspicuous progress in realistic image synthesis with multifarious conditional inputs, while generating diverse yet high-fidelity images remains a grand challenge in conditional image generation. This paper presents a versatile framework for conditional image generation which incorporates the inductive bias of CNNs and powerful sequence modeling of auto-regression that naturally leads to diverse image generation. Instead of independently quantizing the features of multiple domains as in prior research, we design an integrated quantization scheme with a var…
We introduce CAFLOW, a new diverse image-to-image translation model that simultaneously leverages the power of auto-regressive modeling and the modeling efficiency of conditional normalizing flows. We transform the conditioning image into a sequence of latent encodings using a multi-scale normalizing flow and repeat the process for the conditioned image. We model the conditional distribution of the latent encodings by modeling the auto-regressive distributions with an efficient multi-scale normalizing flow, where each conditioning factor affects image synthesis at its respective resolution sca…
Pushing Auto-regressive Models for 3D Shape Generation at Capacity and Scalability
Fudan University（复旦大学）；Fudan ISTBI—ZJNU Algorithm Centre for Brain-inspired Intelligence（复旦大学ISTBI—浙江师范大学类脑智能算法中心）；Tsinghua University（清华大学）；Google（谷歌公司）；Nanjing University（南京大学）；Tencent Youtu lab（腾讯优图实验室）；BAAI (Beijing Academy of Artificial Intelligence)（北京智源人工智能研究院）
X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Models
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）；CPII under InnoHK（创新香港科技促进会）；MThreads AI
SJD++: Improved Speculative Jacobi Decoding for Training-free Acceleration of Discrete Auto-regressive Text-to-Image Generation
The University of Hong Kong（香港大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Peking University（北京大学）；xAI；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）
Error Analyses of Auto-Regressive Video Diffusion Models: A Unified Framework
Nanyang Technological University（南洋理工大学）；National University of Singapore（国立新加坡大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Sea AI Lab（海思人工智能实验室）；Yale University（耶鲁大学）
Multi-modal Auto-regressive Modeling via Visual Words
School of Computer Science, Wuhan University（武汉大学计算机学院）；Shanghai Jiao Tong University（上海交通大学）；School of Information Management, Wuhan University（武汉大学信息管理学院）
Phase recognition plays an essential role for surgical workflow analysis in computer assisted intervention. Transformer, originally proposed for sequential data modeling in natural language processing, has been successfully applied to surgical phase recognition. Existing works based on transformer mainly focus on modeling attention dependency, without introducing auto-regression. In this work, an Auto-Regressive Surgical Transformer, referred as ARST, is first proposed for on-line surgical phase recognition from laparoscopic videos, modeling the inter-phase correlation implicitly by conditiona…
ARDepth: Auto-regressive Monocular Depth Estimation with Progressive Visual Conditioning
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Shenzhen Loop Area Institute（深圳河套学院）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东省大数据分析与处理重点实验室）；Baidu Inc.（百度公司）；LightSpeed Studios, Tencent America（美国光速工作室，腾讯）；School of Computer Science and Artificial Intelligence, Lanzhou University of Technology（兰州理工大学计算机科学与人工智能学院）
High-Fidelity Functional Ultrasound Reconstruction via A Visual Auto-Regressive Framework
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；Department of Computer and Information Science, University of Macau（计算机与信息科学系，澳门大学）
Quality-Aware Language-Conditioned Local Auto-Regressive Anomaly Synthesis and Detection
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（自动化研究所基础模型研究中心，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Objecteye Inc.（Objecteye公司）
SymDrive: Realistic and Controllable Driving Simulator via Symmetric Auto-regressive Online Restoration
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动系统学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；State Key Lab of Intelligent Transportation System, Beihang University（北航智能交通系统国家重点实验室）
Clova AI Research, NAVER Corp.（NAVER公司Clova AI研究中心）
We propose a generic confidence-based approximation that can be plugged in and simplify the auto-regressive generation process with a proved convergence. We first assume that the priors of future samples can be generated in an independently and identically distributed (i.i.d.) manner using an efficient predictor. Given the past samples and future priors, the mother AR model can post-process the priors while the accompanied confidence predictor decides whether the current sample needs a resampling or not. Thanks to the i.i.d. assumption, the post-processing can update each sample in a parallel …
XSpecMesh: Quality-Preserving Auto-Regressive Mesh Generation Acceleration via Multi-Head Speculative Decoding
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（教育部多媒体可信感知与高效计算重点实验室，厦门大学）；Shandong Inspur Database Technology Co.,Ltd（山东浪潮数据库技术有限公司）
GenDoP: Auto-regressive Camera Trajectory Generation as a Director of Photography
Zhejiang University（浙江大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Stanford University（斯坦福大学）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
Visual Computing Institute, RWTH Aachen University（亚琛工业大学视觉计算研究所）
Detecting persons using a 2D LiDAR is a challenging task due to the low information content of 2D range data. To alleviate the problem caused by the sparsity of the LiDAR points, current state-of-the-art methods fuse multiple previous scans and perform detection using the combined scans. The downside of such a backward looking fusion is that all the scans need to be aligned explicitly, and the necessary alignment operation makes the whole pipeline more expensive -- often too expensive for real-world applications. In this paper, we propose a person detection network which uses an alternative st…
