AsyncDiff：通过异步去噪并行化扩散模型
AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising
AI总结：
本文提出AsyncDiff，一种通用即插即用的加速方案，通过将噪声预测模型分割至多设备并利用隐藏状态相似性实现异步去噪，显著降低推理延迟且几乎不损生成质量。
摘要：
扩散模型因其在各种应用中出色的生成能力而受到社区的广泛关注。然而，其典型的多步顺序去噪特性导致了高累积延迟，从而排除了并行计算的可能性。为了解决这个问题，我们引入了AsyncDiff，这是一种通用的即插即用加速方案，可实现跨多个设备的模型并行。我们的方法将庞大的噪声预测模型划分为多个组件，并将每个组件分配给不同的设备。为了打破这些组件之间的依赖链，它利用连续扩散步骤中隐藏状态之间的高度相似性，将传统的顺序去噪转换为异步过程。因此，每个组件都可以在独立的设备上并行计算。所提出的策略显著减少了推理延迟，同时对生成质量的影响最小。具体而言，对于Stable Diffusion v2.1，在四块NVIDIA A5000 GPU上，AsyncDiff实现了2.7倍的加速且性能下降可忽略不计，以及4.0倍的加速且CLIP Score仅略微降低0.38。我们的实验还表明，AsyncDiff可以轻松应用于视频扩散模型，并取得了令人鼓舞的性能。代码可在https://github.com/czg1225/AsyncDiff获取。
英文摘要：
Diffusion models have garnered significant interest from the community for their great generative ability across various applications. However, their typical multi-step sequential-denoising nature gives rise to high cumulative latency, thereby precluding the possibilities of parallel computation. To address this, we introduce AsyncDiff, a universal and plug-and-play acceleration scheme that enables model parallelism across multiple devices. Our approach divides the cumbersome noise prediction model into multiple components, assigning each to a different device. To break the dependency chain between these components, it transforms the conventional sequential denoising into an asynchronous process by exploiting the high similarity between hidden states in consecutive diffusion steps. Consequently, each component is facilitated to compute in parallel on separate devices. The proposed strategy significantly reduces inference latency while minimally impacting the generative quality. Specifically, for the Stable Diffusion v2.1, AsyncDiff achieves a 2.7x speedup with negligible degradation and a 4.0x speedup with only a slight reduction of 0.38 in CLIP Score, on four NVIDIA A5000 GPUs. Our experiments also demonstrate that AsyncDiff can be readily applied to video diffusion models with encouraging performances. The code is available at https://github.com/czg1225/AsyncDiff.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
