Transformer-based diffusion models have achieved significant advancements across a variety of generative tasks. However, producing high-quality outputs typically necessitates large transformer models, which result in substantial training and inference overhead. In this work, we investigate an alternative approach involving multiple experts for denoising, and introduce Remix-DiT, a novel method designed to enhance output quality at a low cost. The goal of Remix-DiT is to craft N diffusion experts for different denoising timesteps, yet without the need for expensive training of N independent models. To achieve this, Remix-DiT employs K basis models (where K < N) and utilizes learnable mixing coefficients to adaptively craft expert models. This design offers two significant advantages: first, although the total model size is increased, the model produced by the mixing operation shares the same architecture as a plain model, making the overall model as efficient as a standard diffusion transformer. Second, the learnable mixing adaptively allocates model capacity across timesteps, thereby effectively improving generation quality. Experiments conducted on the ImageNet dataset demonstrate that Remix-DiT achieves promising results compared to standard diffusion transformers and other multiple-expert methods. The code is available at https://github.com/VainF/Remix-DiT.
Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioning
Tongji University(同济大学);The Ohio State University(俄亥俄州立大学);Georgia Institute of Technology(佐治亚理工学院);Shandong Academy of Sciences(山东省科学院);Macquarie University(麦考瑞大学)
Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Acceleration
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);National Cheng-Kung University(国立成功大学)
针对 DiT 推理计算昂贵的问题,本文提出训练后即插即用的 SDTM,依据“先结构后细节”先验动态合并冗余视觉 token,实现 1.55 倍加速且几乎不损失图像质量。
BWCache: Accelerating Video Diffusion Transformers through Block-Wise Caching
School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China(人工智能学院,北京师范大学,北京);Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai 519087, China(人工智能与未来网络研究院,北京师范大学,珠海)
DiTango: Cost-Effective Parallel Diffusion Generation with Selective Attention State Reuse
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);University of Science and Technology Beijing(北京科技大学);PJlab(PJ实验室)
DiT as Real-Time Rerenderer: Streaming Video Stylization with Autoregressive Diffusion Transformer
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);XMax.AI Ltd.(XMax人工智能有限公司);Nanyang Technological University(南洋理工大学)
TunerDiT: Training-free Progressive Steering of Diffusion Transformer for Multi-Event Video Generation
Ludwig Maximilian University of Munich(慕尼黑路德维希-马克西米利安大学);Technical University of Munich(慕尼黑技术大学);MCML;University of Hamburg(汉堡大学);Huawei European Research Institute(华为欧洲研究院)
Generative Neural Video Compression via Video Diffusion Prior
School of Information and Communication Engineering, Communication University of China(信息与通信工程学院,通信大学);School of Computer Science, Peking University(计算机学院,北京大学)
EndoCoT: Scaling Endogenous Chain-of-Thought Reasoning in Diffusion Models
Shanghai AI Laboratory(上海人工智能实验室);Xi’an Jiaotong University(西安交通大学);University of Science and Technology of China(中国科学技术大学);Shanghai Jiaotong University(上海交通大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
Memory-Efficient Training-Free Acceleration of Diffusion Transformers with BaryCache
College of Electronics and Information Engineering, Shenzhen University(深圳大学电子与信息工程学院);School of Artificial Intelligence, Shenzhen University(深圳大学人工智能学院);College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机与软件学院)
Timestep-Aware Block Masking for Efficient Diffusion Model Inference
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);School of Data Science, Fudan University(复旦大学数据科学学院)