In this paper, we explore a new generative approach for learning visual representations. Our method, DARL, employs a decoder-only Transformer to predict image patches autoregressively. We find that training with Mean Squared Error (MSE) alone leads to strong representations. To enhance the image generation ability, we replace the MSE loss with the diffusion objective by using a denoising patch decoder. We show that the learned representation can be improved by using tailored noise schedules and longer training in larger models. Notably, the optimal schedule differs significantly from the typical ones used in standard image diffusion models. Overall, despite its simple architecture, DARL delivers performance remarkably close to state-of-the-art masked prediction models under the fine-tuning protocol. This marks an important step towards a unified model capable of both visual perception and generation, effectively combining the strengths of autoregressive and denoising diffusion models.
Image captioning task has been extensively researched by previous work. However, limited experiments focus on generating captions based on non-autoregressive text decoder. Inspired by the recent success of the denoising diffusion model on image synthesis tasks, we apply denoising diffusion probabilistic models to text generation in image captioning tasks. We show that our CLIP-Diffusion-LM is capable of generating image captions using significantly fewer inference steps than autoregressive models. On the Flickr8k dataset, the model achieves 0.1876 BLEU-4 score. By training on the combined Flic…
In-Context Forcing: Uncovering Context Effects in Autoregressive Video Diffusion
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Tencent Youtu Lab（腾讯云图实验室）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
Motion-Aware Caching for Efficient Autoregressive Video Generation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，中国教育部，厦门大学）；Max Planck Institute for Intelligent Systems（智能系统马克斯·普朗克研究所）；ELLIS Institute Tübingen（图宾根ELLIS研究所）；Tübingen AI Center（图宾根人工智能中心）
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）；ByteDance（字节跳动）
Continuous Speculative Decoding for Autoregressive Image Generation
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统实验室）；JD.COM Inc（京东集团股份有限公司）；China Tower Corporation Limited（中国铁塔股份有限公司）
Hierarchical Denoising For Multi-Step Visual Reasoning
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机科学学院多媒体信息处理技术国家重点实验室）；The Hong Kong University of Science and Technology（香港科技大学）；Beihang University（北京航空航天大学）；Fuzhou University（福州大学）；Muka Robotics（木卡机器人）
We implement stacked denoising autoencoders, a class of neural networks that are capable of learning powerful representations of high dimensional data. We describe stochastic gradient descent for unsupervised training of autoencoders, as well as a novel genetic algorithm based approach that makes use of gradient information. We analyze the performance of both optimization algorithms and also the representation learning ability of the autoencoder when it is trained on standard image classification datasets.
Denoising diffusion probabilistic models (DDPMs) (Ho et al. 2020) have shown impressive results on image and waveform generation in continuous state spaces. Here, we introduce Discrete Denoising Diffusion Probabilistic Models (D3PMs), diffusion-like generative models for discrete data that generalize the multinomial diffusion model of Hoogeboom et al. 2021, by going beyond corruption processes with uniform transition probabilities. This includes corruption with transition matrices that mimic Gaussian kernels in continuous space, matrices based on nearest neighbors in embedding space, and matri…
DiffVC: A Non-autoregressive Framework Based on Diffusion Model for Video Captioning
School of Software, Northwestern Polytechnical University（西北工业大学软件学院）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；Beijing University Of Technology（北京工业大学）
Free-form inpainting is the task of adding new content to an image in the regions specified by an arbitrary binary mask. Most existing approaches train for a certain distribution of masks, which limits their generalization capabilities to unseen mask types. Furthermore, training with pixel-wise and perceptual losses often leads to simple textural extensions towards the missing areas instead of semantically meaningful generation. In this work, we propose RePaint: A Denoising Diffusion Probabilistic Model (DDPM) based inpainting approach that is applicable to even extreme masks. We employ a pret…
Context plays an important role in human language understanding, thus it may also be useful for machines learning vector representations of language. In this paper, we explore an asymmetric encoder-decoder structure for unsupervised context-based sentence representation learning. We carefully designed experiments to show that neither an autoregressive decoder nor an RNN decoder is required. After that, we designed a model which still keeps an RNN as the encoder, while using a non-autoregressive convolutional decoder. We further combine a suite of effective designs to significantly improve mode…
Not All Frames Deserve Full Computation: Accelerating Autoregressive Video Generation via Selective Computation and Predictive Extrapolation
Institute of Artificial Intelligence and Future Networks, Beijing Normal University（北京师范大学人工智能与未来网络研究院）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Shenzhen University of Advanced Technology（深圳理工大学）
Arch-VQ: Discrete Architecture Representation Learning with Autoregressive Priors
AI, Optimization and Pattern Recognition Research Group, Dept. of Mechanical Eng., University of Melbourne, Australia（墨尔本大学人工智能、优化与模式识别研究组）；School of Electronics and Computer Science, University of Southampton, UK（南安普顿大学电子与计算机科学学院）；KINDI Computing Research Center, College of Engineering, Qatar University, Doha, Qatar（卡塔尔大学工程学院KINDI计算研究中心）
Macau University of Science and Technology（澳门科技大学）；Communication University of Zhejiang（浙江传媒学院）；Ningbo University of Finance & Economics（宁波财经学院）
Speech-driven gesture synthesis is a field of growing interest in virtual human creation. However, a critical challenge is the inherent intricate one-to-many mapping between speech and gestures. Previous studies have explored and achieved significant progress with generative models. Notwithstanding, most synthetic gestures are still vastly less natural. This paper presents DiffMotion, a novel speech-driven gesture synthesis architecture based on diffusion models. The model comprises an autoregressive temporal encoder and a denoising diffusion probability Module. The encoder extracts the tempor…
