arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.19481cs.CV
收藏

DistriFusion:面向高分辨率扩散模型的分布式并行推理

DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models

MIT(麻省理工学院);Princeton(普林斯顿大学);Lepton AI(轻子人工智能公司);NVIDIA(英伟达公司)

Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Ming-Yu Liu, Kai Li, Song Han

AI总结:

针对高分辨率扩散模型推理延迟高的问题,提出DistriFusion分布式并行方法,通过错位块并行复用前序特征实现异步通信,在多GPU上实现显著加速且无质量损失。

摘要:

扩散模型在合成高质量图像方面已取得巨大成功。然而,使用扩散模型生成高分辨率图像仍然颇具挑战,因为其计算成本极高,导致交互类应用的延迟高到令人无法接受。本文提出DistriFusion来解决这一问题,该方法利用多GPU间的并行性提升效率。我们的方法将模型输入拆分为多个图像块,将每个块分配给一个GPU。但直接实现这类算法会破坏图像块间的交互,降低生成保真度,而加入这类交互又会带来巨大的通信开销。为破解这一困境,我们观察到相邻扩散步骤的输入具有高度相似性,由此提出错位块并行策略:该策略利用扩散过程的时序特性,复用前一时间步预计算的特征图,为当前步骤提供上下文信息。因此,我们的方法支持异步通信,可与计算过程流水线化。大量实验表明,该方法可应用于最新的Stable Diffusion XL模型且无质量下降,在8张NVIDIA A100上相比单卡实现了最高6.1倍的加速比。我们的代码已公开在https://github.com/mit-han-lab/distrifuser。

英文摘要:

Diffusion models have achieved great success in synthesizing high-quality images. However, generating high-resolution images with diffusion models is still challenging due to the enormous computational costs, resulting in a prohibitive latency for interactive applications. In this paper, we propose DistriFusion to tackle this problem by leveraging parallelism across multiple GPUs. Our method splits the model input into multiple patches and assigns each patch to a GPU. However, naively implementing such an algorithm breaks the interaction between patches and loses fidelity, while incorporating such an interaction will incur tremendous communication overhead. To overcome this dilemma, we observe the high similarity between the input from adjacent diffusion steps and propose displaced patch parallelism, which takes advantage of the sequential nature of the diffusion process by reusing the pre-computed feature maps from the previous timestep to provide context for the current step. Therefore, our method supports asynchronous communication, which can be pipelined by computation. Extensive experiments show that our method can be applied to recent Stable Diffusion XL with no quality degradation and achieve up to a 6.1$\times$ speedup on eight NVIDIA A100s compared to one. Our code is publicly available at https://github.com/mit-han-lab/distrifuser.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑