Recent advances in pretraining general foundation models have significantly improved performance across diverse downstream tasks. While autoregressive (AR) generative models like GPT have revolutionized NLP, most visual generative pretraining methods still rely on BERT-style masked modeling, which often disregards the temporal information essential for video analysis. The few existing autoregressive visual pretraining methods suffer from issues such as inaccurate semantic localization and poor generation quality, leading to poor semantics. In this work, we propose NExT-Vid, a novel autoregressive visual generative pretraining framework that utilizes masked next-frame prediction to jointly model images and videos. NExT-Vid introduces a context-isolated autoregressive predictor to decouple semantic representation from target decoding, and a conditioned flow-matching decoder to enhance generation quality and diversity. Through context-isolated flow-matching pretraining, our approach achieves strong representations. Extensive experiments on large-scale pretrained models demonstrate that our proposed method consistently outperforms previous generative pretraining methods for visual representation learning via attentive probing in downstream classification.
GLOBER: Coherent Non-autoregressive Video Generation via GLOBal Guided Video DecodER
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantization
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
DiffVC: A Non-autoregressive Framework Based on Diffusion Model for Video Captioning
School of Software, Northwestern Polytechnical University（西北工业大学软件学院）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；Beijing University Of Technology（北京工业大学）
We make a minimal, but very effective alteration to the VAE model. This is about a drop-in replacement for the (sample-dependent) approximate posterior to change it from the standard white Gaussian with diagonal covariance to the first-order autoregressive Gaussian. We argue that this is a more reasonable choice to adopt for natural signals like images, as it does not force the existing correlation in the data to disappear in the posterior. Moreover, it allows more freedom for the approximate posterior to match the true posterior. This allows for the repararametrization trick, as well as the K…
Google Research（谷歌研究院）；CUHK（香港中文大学）；Google Cloud AI（谷歌云人工智能）
We propose Stratified Image Transformer(StraIT), a pure non-autoregressive(NAR) generative model that demonstrates superiority in high-quality image synthesis over existing autoregressive(AR) and diffusion models(DMs). In contrast to the under-exploitation of visual characteristics in existing vision tokenizer, we leverage the hierarchical nature of images to encode visual tokens into stratified levels with emergent properties. Through the proposed image stratification that obtains an interlinked token pair, we alleviate the modeling difficulty and lift the generative power of NAR models. Our …
ARMO: Autoregressive Rigging for Multi-Category Objects
Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shanghai AI Laboratory（上海人工智能实验室）
Spatiotemporal Satellite Image Downscaling with Transfer Encoders and Autoregressive Generative Models
University of Toronto（多伦多大学）；Harvard T.H. Chan School of Public Health（哈佛大学T.H. Chan公共卫生学院）；Harvard Medical School（哈佛医学院）；VA Healthcare System Boston, U.S. Department of Veterans Affairs（美国退伍军人事务部波士顿医疗系统）
We present the Pathways Autoregressive Text-to-Image (Parti) model, which generates high-fidelity photorealistic images and supports content-rich synthesis involving complex compositions and world knowledge. Parti treats text-to-image generation as a sequence-to-sequence modeling problem, akin to machine translation, with sequences of image tokens as the target outputs rather than text tokens in another language. This strategy can naturally tap into the rich body of prior work on large language models, which have seen continued advances in capabilities and performance through scaling data and …
BioVid: Autoregressive Video Generation with Biological Behavior Semantic Comprehension
Department of Electrical Engineering, National Taiwan Ocean University（国立台湾海洋大学电子工程系）；AI research center, National Taiwan Ocean University（国立台湾海洋大学人工智能研究中心）
In-Context Forcing: Uncovering Context Effects in Autoregressive Video Diffusion
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Tencent Youtu Lab（腾讯云图实验室）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；College of Computer science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Computer Science Department, Watson School, The State University of New York, Binghamton（纽约州立大学宾汉姆顿分校沃森学院计算机科学系）；School of Information Engineering, Zhejiang A&F University（浙江农林大学信息工程学院）
Human motion prediction, which aims at predicting future human skeletons given the past ones, is a typical sequence-to-sequence problem. Therefore, extensive efforts have been continued on exploring different RNN-based encoder-decoder architectures. However, by generating target poses conditioned on the previously generated ones, these models are prone to bringing issues such as error accumulation problem. In this paper, we argue that such issue is mainly caused by adopting autoregressive manner. Hence, a novel Non-auToregressive Model (NAT) is proposed with a complete non-autoregressive decod…
Autoregressive language modeling (ALM) have been successfully used in self-supervised pre-training in Natural language processing (NLP). However, this paradigm has not achieved comparable results with other self-supervised approach in computer vision (e.g., contrastive learning, mask image modeling). In this paper, we try to find the reason why autoregressive modeling does not work well on vision tasks. To tackle this problem, we fully analyze the limitation of visual autoregressive methods and proposed a novel stochastic autoregressive image modeling (named SAIM) by the two simple designs. Fi…
Seg-VAR: Image Segmentation with Visual Autoregressive Modeling
The University of Hong Kong（香港大学）；Insta360；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Innovation Institute（上海创新研究院）；SenseTime Research（商汤科技研究院）
Generative modeling of high-dimensional data is a key problem in machine learning. Successful approaches include latent variable models and autoregressive models. The complementary strengths of these approaches, to model global and local image statistics respectively, suggest hybrid models that encode global image structure into latent variables while autoregressively modeling low level detail. Previous approaches to such hybrid models restrict the capacity of the autoregressive decoder to prevent degenerate models that ignore the latent variables and only rely on autoregressive modeling. Our …
Tsinghua University（清华大学）；University of Chinese Academy of Sciences（中国科学院大学）
In this letter, we explored generative image steganography based on autoregressive models. We proposed Pixel-Stega, which implements pixel-level information hiding with autoregressive models and arithmetic coding algorithm. Firstly, one of the autoregressive models, PixelCNN++, is utilized to produce explicit conditional probability distribution of each pixel. Secondly, secret messages are encoded to the selection of pixels through steganographic sampling (stegosampling) based on arithmetic coding. We carried out qualitative and quantitative assessment on gray-scale and colour image datasets. …
Pedestrian trajectory prediction is a challenging task as there are three properties of human movement behaviors which need to be addressed, namely, the social influence from other pedestrians, the scene constraints, and the multimodal (multiroute) nature of predictions. Although existing methods have explored these key properties, the prediction process of these methods is autoregressive. This means they can only predict future locations sequentially. In this paper, we present NAP, a non-autoregressive method for trajectory prediction. Our method comprises specifically designed feature encode…
Future frame prediction has been approached through two primary methods: autoregressive and non-autoregressive. Autoregressive methods rely on the Markov assumption and can achieve high accuracy in the early stages of prediction when errors are not yet accumulated. However, their performance tends to decline as the number of time steps increases. In contrast, non-autoregressive methods can achieve relatively high performance but lack correlation between predictions for each time step. In this paper, we propose an Implicit Stacked Autoregressive Model for Video Prediction (IAM4VP), which is an …
École Polytechnique Fédéral de Lausanne（洛桑联邦理工学院）；Idiap Research Institute（IDIAP研究所）
We propose to leverage Transformer architectures for non-autoregressive human motion prediction. Our approach decodes elements in parallel from a query sequence, instead of conditioning on previous predictions such as instate-of-the-art RNN-based approaches. In such a way our approach is less computational intensive and potentially avoids error accumulation to long term elements in the sequence. In that context, our contributions are fourfold: (i) we frame human motion prediction as a sequence-to-sequence problem and propose a non-autoregressive Transformer to infer the sequences of poses in p…
University of British Columbia（不列颠哥伦比亚大学）；Massachusetts Institute of Technology（麻省理工学院）；South China University of Technology（华南理工大学）；Google Research（谷歌研究院）；Tsinghua University（清华大学）；Vector Institute for AI（向量人工智能研究所）
Inspired by the success of self-supervised autoregressive representation learning in natural language (GPT and its variants), and advances in recent visual architecture design with Vision Transformers (ViTs), in this paper, we explore the effect various design choices have on the success of applying such training strategies for visual feature learning. Specifically, we introduce a novel strategy that we call Random Segments with Autoregressive Coding (RandSAC). In RandSAC, we group patch representations (image tokens) into hierarchically arranged segments; within each segment, tokens are predi…
