We introduce TransDiff, the first image generation model that marries Autoregressive (AR) Transformer with diffusion models. In this joint modeling framework, TransDiff encodes labels and images into high-level semantic features and employs a diffusion model to estimate the distribution of image samples. On the ImageNet 256x256 benchmark, TransDiff significantly outperforms other image generation models based on standalone AR Transformer or diffusion models. Specifically, TransDiff achieves a Frechet Inception Distance (FID) of 1.61 and an Inception Score (IS) of 293.4, and further provides x2 faster inference latency compared to state-of-the-art methods based on AR Transformer and x112 faster inference compared to diffusion-only models. Furthermore, building on the TransDiff model, we introduce a novel image generation paradigm called Multi-Reference Autoregression (MRAR), which performs autoregressive generation by predicting the next image. MRAR enables the model to reference multiple previously generated images, thereby facilitating the learning of more diverse representations and improving the quality of generated images in subsequent iterations. By applying MRAR, the performance of TransDiff is improved, with the FID reduced from 1.61 to 1.42. We expect TransDiff to open up a new frontier in the field of image generation.
SAMPO:Scale-wise Autoregression with Motion PrOmpt for generative world models
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家重点实验室);National Engineering Research Center for Visual Information and Applications(视觉信息与应用国家工程研究中心);Institute of Artificial Intelligence and Robotics(人工智能与机器人研究院);Xi’an Jiaotong University(西安交通大学);University of Illinois at Chicago(伊利诺伊大学香槟分校);Amazon.com, Inc(亚马逊公司)
GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression
University of Rochester(罗切斯特大学);University of Tokyo(东京大学);University of California Santa Cruz(加州大学圣克鲁兹分校);University of California Los Angeles(加州大学洛杉矶分校);Meta
Marrying Text-to-Motion Generation with Skeleton-Based Action Recognition
School of Cyber Science and Engineering, Southeast University, Nanjing, China(东南大学计算机科学与工程学院,南京,中国);School of Computer Science and Engineering, Southeast University, Nanjing, China(东南大学计算机科学与工程学院,南京,中国)
NOVA: Next-step Open-Vocabulary Autoregression for 3D Multi-Object Tracking in Autonomous Driving
College of Mechanical and Vehicle Engineering, Hunan University(湖南大学机械与运载工程学院);State Key Laboratory of Intelligent Autonomous Systems, Tongji University(同济大学智能自主系统国家重点实验室)
AlignVAR: Towards Globally Consistent Visual Autoregression for Image Super-Resolution
University of Electronic Science and Technology of China(电子科技大学);Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province(四川省 ubiquitous intelligence and trusted services 重点实验室)
Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction
The State Key Lab of Brain-Machine Intelligence, Zhejiang University, China(脑机智能国家重点实验室,浙江大学,中国);ReLER, CCAI, College of Artificial Intelligence, Zhejiang University, China(ReLER、中国人工智能学会、人工智能学院、浙江大学、中国)