Conversational scenarios are very common in real-world settings, yet existing co-speech motion synthesis approaches often fall short in these contexts, where one person's audio and gestures will influence the other's responses. Additionally, most existing methods rely on offline sequence-to-sequence frameworks, which are unsuitable for online applications. In this work, we introduce an audio-driven, auto-regressive system designed to synthesize dynamic movements for two characters during a conversation. At the core of our approach is a diffusion-based full-body motion synthesis model, which is conditioned on the past states of both characters, speech audio, and a task-oriented motion trajectory input, allowing for flexible spatial control. To enhance the model's ability to learn diverse interactions, we have enriched existing two-person conversational motion datasets with more dynamic and interactive motions. We evaluate our system through multiple experiments to show it outperforms across a variety of tasks, including single and two-person co-speech motion generation, as well as interactive motion generation. To the best of our knowledge, this is the first system capable of generating interactive full-body motions for two characters from speech in an online manner.
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding
The University of Hong Kong（香港大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；CUHK（香港中文大学）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Infinigence AI（智源人工智能）
University of Waterloo（滑铁卢大学）；Alibaba Group（阿里巴巴集团）；Vector Institute（向量研究所）
Conditioned diffusion models have demonstrated state-of-the-art text-to-image synthesis capacity. Recently, most works focus on synthesizing independent images; While for real-world applications, it is common and necessary to generate a series of coherent images for story-stelling. In this work, we mainly focus on story visualization and continuation tasks and propose AR-LDM, a latent diffusion model auto-regressively conditioned on history captions and generated images. Moreover, AR-LDM can generalize to new characters through adaptation. To our best knowledge, this is the first work successf…
Nanyang Technological University（南洋理工大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Amazon（亚马逊）
Deep generative models have achieved conspicuous progress in realistic image synthesis with multifarious conditional inputs, while generating diverse yet high-fidelity images remains a grand challenge in conditional image generation. This paper presents a versatile framework for conditional image generation which incorporates the inductive bias of CNNs and powerful sequence modeling of auto-regression that naturally leads to diverse image generation. Instead of independently quantizing the features of multiple domains as in prior research, we design an integrated quantization scheme with a var…
X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Models
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）；CPII under InnoHK（创新香港科技促进会）；MThreads AI
Pushing Auto-regressive Models for 3D Shape Generation at Capacity and Scalability
Fudan University（复旦大学）；Fudan ISTBI—ZJNU Algorithm Centre for Brain-inspired Intelligence（复旦大学ISTBI—浙江师范大学类脑智能算法中心）；Tsinghua University（清华大学）；Google（谷歌公司）；Nanjing University（南京大学）；Tencent Youtu lab（腾讯优图实验室）；BAAI (Beijing Academy of Artificial Intelligence)（北京智源人工智能研究院）
We introduce CAFLOW, a new diverse image-to-image translation model that simultaneously leverages the power of auto-regressive modeling and the modeling efficiency of conditional normalizing flows. We transform the conditioning image into a sequence of latent encodings using a multi-scale normalizing flow and repeat the process for the conditioned image. We model the conditional distribution of the latent encodings by modeling the auto-regressive distributions with an efficient multi-scale normalizing flow, where each conditioning factor affects image synthesis at its respective resolution sca…
Error Analyses of Auto-Regressive Video Diffusion Models: A Unified Framework
Nanyang Technological University（南洋理工大学）；National University of Singapore（国立新加坡大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Sea AI Lab（海思人工智能实验室）；Yale University（耶鲁大学）
SJD++: Improved Speculative Jacobi Decoding for Training-free Acceleration of Discrete Auto-regressive Text-to-Image Generation
The University of Hong Kong（香港大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Peking University（北京大学）；xAI；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）
Institute of Medical Technology, Peking University Health Science Center, Peking University（北京大学医学部医学技术研究所）；Baidu VIS（百度VIS）；Biomedical Engineering Department, College of Future Technology, Peking University（未来技术学院生物医学工程系）；National Biomedical Imaging Center, Peking University（北京大学国家生物医学成像中心）；Tsinghua University（清华大学）
The image captioning task is typically realized by an auto-regressive method that decodes the text tokens one by one. We present a diffusion-based captioning model, dubbed the name DDCap, to allow more decoding flexibility. Unlike image generation, where the output is continuous and redundant with a fixed length, texts in image captions are categorical and short with varied lengths. Therefore, naively applying the discrete diffusion model to text decoding does not work well, as shown in our experiments. To address the performance gap, we propose several key techniques including best-first infe…
Generative adversarial networks (GANs) have been indicated their superiority in usage of the real-time speech synthesis. Nevertheless, most of them make use of deep convolutional layers as their backbone, which may cause the absence of previous signal information. However, the generation of speech signals invariably require preceding waveform samples in its reconstruction, as the lack of this can lead to artifacts in generated speech. To address this conflict, in this paper, we propose an improved model: a post auto-regressive (AR) GAN vocoder with a self-attention layer, which merging self-at…
EditGen: Harnessing Cross-Attention Control for Instruction-Based Auto-Regressive Audio Editing
Department of Informatics and Telecommunications, UoA NCSR "Demokritos"（信息与电信系，雅典国家科研中心"Demokritos"）；School of Electrical and Computer Engineering, NTUA（电气与计算机工程学院，国家技术研究院）
Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems
Digital Trust Centre, Nanyang Technological University（南洋理工大学数字信托中心）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
Recent neural text-to-speech (TTS) models with fine-grained latent features enable precise control of the prosody of synthesized speech. Such models typically incorporate a fine-grained variational autoencoder (VAE) structure, extracting latent features at each input token (e.g., phonemes). However, generating samples with the standard VAE prior often results in unnatural and discontinuous speech, with dramatic prosodic variation between tokens. This paper proposes a sequential prior in a discrete latent space which can generate more naturally sounding samples. This is accomplished by discreti…
SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation
Beihang University（北京航空航天大学）；Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Harbin Institute of Technology（哈尔滨工业大学）；CPII under InnoHK（InnoHK下的CPII）
