Designed to learn long-range interactions on sequential data, transformers continue to show state-of-the-art results on a wide variety of tasks. In contrast to CNNs, they contain no inductive bias that prioritizes local interactions. This makes them expressive, but also computationally infeasible for long sequences, such as high-resolution images. We demonstrate how combining the effectiveness of the inductive bias of CNNs with the expressivity of transformers enables them to model and thereby synthesize high-resolution images. We show how to (i) use CNNs to learn a context-rich vocabulary of image constituents, and in turn (ii) utilize transformers to efficiently model their composition within high-resolution images. Our approach is readily applied to conditional synthesis tasks, where both non-spatial information, such as object classes, and spatial information, such as segmentations, can control the generated image. In particular, we present the first results on semantically-guided synthesis of megapixel images with transformers and obtain the state of the art among autoregressive models on class-conditional ImageNet. Code and pretrained models can be found at https://github.com/CompVis/taming-transformers .
MV-Performer: Taming Video Diffusion Model for Faithful and Synchronized Multi-view Performer Synthesis
Great Bay University（大亚湾大学）；School of Artificial Intelligence, Shenzhen University（深圳大学人工智能学院）；Guangdong Provincial Key Laboratory of Future Networks of Intelligence（广东省未来网络智能化重点实验室）
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis
Peking University（北京大学）；Peng Cheng Laboratory（鹏城实验室）；The Chinese University of Hong Kong（香港中文大学）；Tencent AI Lab（腾讯AI Lab）；ARC Lab, Tencent PCG（腾讯PCG ARC实验室）；Hong Kong University of Science and Technology（香港科技大学）；Monash University（莫纳什大学）
UniGP: Taming Diffusion Transformer for Prior-Preserved Unified Generation and Perception
The Hong Kong University of Science and Technology（香港科学与技术大学）；DAMO Academy, Alibaba Group（阿里云达摩院）；Hupan Lab（虎扑实验室）；Zhejiang University（浙江大学）；Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）；Zeekr Automobile R&D Co., Ltd.（小鹏汽车研发有限公司）
UniMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation
The Hong Kong University of Science and Technology（香港科学与技术大学）；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）；Tencent（腾讯）；National University of Singapore（新加坡国立大学）
Taming Generative Synthetic Data for X-ray Prohibited Item Detection
Institute of Information Science, Beijing Jiaotong University（信息科学学院，北京交通大学）；Faculty of Data Science, City University of Macau（数据科学学院，澳门城市大学）；Nuctech Company Limited（NuTech公司）
VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Huazhong University of Science and Technology（华中科技大学）；Shenzhen Loop Area Institute（深圳河套学院）；University of Science and Technology of China（中国科学技术大学）
TReFT: Taming Rectified Flow Models For One-Step Image Translation
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Beihang University（北航）；Tianjin University（天津大学）；CreateAI
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Peking University（北京大学）
Automatic generation of high-quality Chinese fonts from a few online training samples is a challenging task, especially when the amount of samples is very small. Existing few-shot font generation methods can only synthesize low-resolution glyph images that often possess incorrect topological structures or/and incomplete strokes. To address the problem, this paper proposes FontTransformer, a novel few-shot learning model, for high-resolution Chinese glyph image synthesis by using stacked Transformers. The key idea is to apply the parallel Transformer to avoid the accumulation of prediction erro…
VolDiT: Controllable Volumetric Medical Image Synthesis with Diffusion Transformers
Institute for Artificial Intelligence in Cardiovascular Medicine, Medical Faculty of Heidelberg University, Heidelberg University（海德堡大学医学院心血管医学人工智能研究所）；Department of Cardiology, Angiology, Pneumology, Heidelberg University Hospital（海德堡大学医院心脏病学、血管学、肺病学系）；DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim（德国心血管研究中心（DZHK）海德堡/曼海姆合作站点）；University Medical Center of the Johannes Gutenberg-University Mainz（美因茨约翰内斯·古腾堡大学医学中心）
Transformer is eminently suitable for auto-regressive image synthesis which predicts discrete value from the past values recursively to make up full image. Especially, combined with vector quantised latent representation, the state-of-the-art auto-regressive transformer displays realistic high-resolution images. However, sampling the latent code from discrete probability distribution makes the output unpredictable. Therefore, it requires to generate lots of diverse samples to acquire desired outputs. To alleviate the process of generating lots of samples repetitively, in this article, we propo…
Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Baidu（百度）；Tokyo Institute of Technology（东京工业大学）；Tsinghua University（清华大学）；University of Science and Technology of China（中国科学技术大学）；Nanyang Technological University（南洋理工大学）
MMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis
South China University of Technology（华南理工大学）；JD Explore Academy（京东探索研究院）；Nanyang Technological University（南洋理工大学）；Shanghai Jiao Tong University（上海交通大学）
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China（中国教育部机器智能与先进计算重点实验室）；Microsoft Research Asia（微软亚洲研究院）；University of Tokyo（东京大学）
We present a new perspective of achieving image synthesis by viewing this task as a visual token generation problem. Different from existing paradigms that directly synthesize a full image from a single input (e.g., a latent code), the new formulation enables a flexible local manipulation for different image regions, which makes it possible to learn content-aware and fine-grained style control for image synthesis. Specifically, it takes as input a sequence of latent tokens to predict the visual tokens for synthesizing an image. Under this perspective, we propose a token-based generator (i.e.,T…
We present an end-to-end Transformer based Latent Diffusion model for image synthesis. On the ImageNet class conditioned generation task we show that a Transformer based Latent Diffusion model achieves a 14.1FID which is comparable to the 13.1FID score of a UNet based architecture. In addition to showing the application of Transformer models for Diffusion based image synthesis this simplification in architecture allows easy fusion and modeling of text and image data. The multi-head attention mechanism of Transformers enables simplified interaction between the image and text features which remo…
DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Model
Horizon Robotics；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；Institute of Automation Chinese Academy of Sciences（中国科学院自动化研究所）；The University of Hong Kong（香港大学）；University of Pennsylvania（宾夕法尼亚大学）；HKUST(GZ)（香港科技大学（广州））；University of Chinese Academy of Sciences（中国科学院大学）；HKUST（香港科技大学）
Existing keyframe-based motion synthesis mainly focuses on the generation of cyclic actions or short-term motion, such as walking, running, and transitions between close postures. However, these methods will significantly degrade the naturalness and diversity of the synthesized motion when dealing with complex and impromptu movements, e.g., dance performance and martial arts. In addition, current research lacks fine-grained control over the generated motion, which is essential for intelligent human-computer interaction and animation creation. In this paper, we propose a novel keyframe-based mo…
Story Visualization is an advanced task of computed vision that targets sequential image synthesis, where the generated samples need to be realistic, faithful to their conditioning and sequentially consistent. Our work proposes a novel architectural and training approach: the Impartial Transformer achieves both text-relevant plausible scenes and sequential consistency utilizing as few trainable parameters as possible. This enhancement is even able to handle synthesis of 'hard' samples with occluded objects, achieving improved evaluation metrics comparing to past approaches.
Conditional image synthesis aims to create an image according to some multi-modal guidance in the forms of textual descriptions, reference images, and image blocks to preserve, as well as their combinations. In this paper, instead of investigating these control signals separately, we propose a new two-stage architecture, M6-UFC, to unify any number of multi-modal controls. In M6-UFC, both the diverse control signals and the synthesized image are uniformly represented as a sequence of discrete tokens to be processed by Transformer. Different from existing two-stage autoregressive approaches suc…
The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
Animating virtual avatars to make co-speech gestures facilitates various applications in human-machine interaction. The existing methods mainly rely on generative adversarial networks (GANs), which typically suffer from notorious mode collapse and unstable training, thus making it difficult to learn accurate audio-gesture joint distributions. In this work, we propose a novel diffusion-based framework, named Diffusion Co-Speech Gesture (DiffGesture), to effectively capture the cross-modal audio-to-gesture associations and preserve temporal coherence for high-fidelity audio-driven co-speech gest…
CardioDiT: Latent Diffusion Transformers for 4D Cardiac MRI Synthesis
Institute for Artificial Intelligence in Cardiovascular Medicine, Medical Faculty of Heidelberg University, Heidelberg University（海德堡大学医学院心血管医学人工智能研究所，海德堡大学）；Department of Cardiology, Angiology, Pneumology, Heidelberg University Hospital（海德堡大学医院心脏病学、血管学、肺病学系）；DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim（德国心血管研究中心（DZHK），海德堡/曼海姆合作站点）；Department of Radiology and Nuclear Medicine, University Medical Center Mannheim（曼海姆大学医学中心放射学与核医学系）；Section for Invasive Cardiology and Electrophysiology, I. Medical Department, Cardiology, Hemostasiology and Intensive Care, University Medical Centre Mannheim（曼海姆大学医学中心第一内科（心脏病学、止血学与重症监护）介入心脏病学与电生理学科）；University Medical Center of the Johannes Gutenberg-University Mainz（美因茨约翰内斯·古腾堡大学医学中心）；Department of Cardiology, Angiology, Hemostasis, and Medical Intensive Care, University Medical Centre Mannheim, Medical Faculty Mannheim, University of Heidelberg（海德堡大学曼海姆医学院曼海姆大学医学中心心脏病学、血管学、止血学与内科重症监护系）
本文提出CardioDiT，一种基于扩散变换器的4D潜在扩散模型，用于合成短轴 cine CMR，通过联合建模空间和时间实现连续心脏动态生成。
DP-aware AdaLN-Zero: Taming Conditioning-Induced Heavy-Tailed Gradients in Differentially Private Diffusion
School of Computer Science（计算机科学学院）；Big Data, Minjiang University, Fuzhou, Fujian, China.（大数据，闽江大学，福州，福建，中国）；School of Information, Renmin University of China, Beijing, China（信息学院，中国人民大学，北京，中国）
Recent advances in visually-induced audio generation are based on sampling short, low-fidelity, and one-class sounds. Moreover, sampling 1 second of audio from the state-of-the-art model takes minutes on a high-end GPU. In this work, we propose a single model capable of generating visually relevant, high-fidelity sounds prompted with a set of frames from open-domain videos in less time than it takes to play it on a single GPU. We train a transformer to sample a new spectrogram from the pre-trained spectrogram codebook given the set of video features. The codebook is obtained using a variant of…
