arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.05628cs.CVcs.AI
收藏

Remix-DiT:用于多专家去噪的混合扩散Transformer

Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising

National University of Singapore(新加坡国立大学)

Gongfan Fang, Xinyin Ma, Xinchao Wang

AI总结:

针对大型扩散Transformer训练推理成本高的问题,提出Remix-DiT方法,通过少量基模型加可学习混合系数构建多时间步去噪专家,在ImageNet上以相当的效率实现了更优的生成质量。

摘要:

基于Transformer的扩散模型已在各类生成任务中取得显著进展。然而,生成高质量输出通常需要大型Transformer模型,这会带来高昂的训练与推理开销。在本研究中,我们探索了一种涉及多专家去噪的替代方案,并提出了Remix-DiT——一种旨在以低成本提升输出质量的新方法。Remix-DiT的目标是为不同去噪时间步构建N个扩散专家,且无需对N个独立模型进行成本高昂的训练。为实现这一目标,Remix-DiT采用K个基模型(其中K<N),并利用可学习的混合系数自适应构建专家模型。该设计具备两大显著优势:其一,尽管总模型规模有所增加,但通过混合操作生成的模型与普通模型共享相同架构,使得整体模型的效率与标准扩散Transformer相当。其二,可学习的混合机制可在不同时间步间自适应分配模型容量,从而有效提升生成质量。在ImageNet数据集上开展的实验表明,与标准扩散Transformer及其他多专家方法相比,Remix-DiT取得了颇具前景的结果。代码可访问https://github.com/VainF/Remix-DiT获取。

英文摘要:

Transformer-based diffusion models have achieved significant advancements across a variety of generative tasks. However, producing high-quality outputs typically necessitates large transformer models, which result in substantial training and inference overhead. In this work, we investigate an alternative approach involving multiple experts for denoising, and introduce Remix-DiT, a novel method designed to enhance output quality at a low cost. The goal of Remix-DiT is to craft N diffusion experts for different denoising timesteps, yet without the need for expensive training of N independent models. To achieve this, Remix-DiT employs K basis models (where K < N) and utilizes learnable mixing coefficients to adaptively craft expert models. This design offers two significant advantages: first, although the total model size is increased, the model produced by the mixing operation shares the same architecture as a plain model, making the overall model as efficient as a standard diffusion transformer. Second, the learnable mixing adaptively allocates model capacity across timesteps, thereby effectively improving generation quality. Experiments conducted on the ImageNet dataset demonstrate that Remix-DiT achieves promising results compared to standard diffusion transformers and other multiple-expert methods. The code is available at https://github.com/VainF/Remix-DiT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑