arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.16486cs.CVcs.AI
收藏

PanGu-Draw:利用时间解耦训练与可复用Coop-Diffusion推进资源高效的文本到图像合成

PanGu-Draw: Advancing Resource-Efficient Text-to-Image Synthesis with Time-Decoupled Training and Reusable Coop-Diffusion

Huawei(华为)

Guansong Lu, Yuanfan Guo, Jianhua Han, Minzhe Niu, Yihan Zeng, Songcen Xu, Zeyi Huang, Zhao Zhong, Wei Zhang, Hang Xu

AI总结:

PanGu-Draw提出时间解耦训练策略和Coop-Diffusion算法,实现资源高效的文本到图像及多控制合成,大幅降低数据与训练成本,并支持任意分辨率。

摘要:

当前的大规模扩散模型在条件图像合成方面取得了巨大飞跃,能够解释文本、人体姿态和边缘等多种线索。然而,它们对大量计算资源和广泛数据收集的依赖仍然是一个瓶颈。另一方面,现有扩散模型的集成,每个模型专门针对不同的控制并在独特的潜在空间中运行,由于图像分辨率和潜在空间嵌入结构不兼容,对它们的联合使用构成了挑战。针对这些限制,我们提出了“PanGu-Draw”,一种新颖的潜在扩散模型,专为资源高效的文本到图像合成而设计,能够灵活地容纳多种控制信号。我们首先提出了一种资源高效的时间解耦训练策略,将单一的文本到图像模型拆分为结构生成器和纹理生成器。每个生成器使用一种最大化数据利用率和计算效率的方案进行训练,将数据准备削减了48%,并将训练资源减少了51%。其次,我们引入了“Coop-Diffusion”,一种算法,能够在统一的去噪过程中实现具有不同潜在空间和预定义分辨率的各种预训练扩散模型的协作使用。这使得在任意分辨率下进行多控制图像合成成为可能,无需额外数据或重新训练。对PanGu-Draw的实证验证显示了其在文本到图像和多控制图像生成方面的卓越能力,为未来模型训练效率和生成多样性指明了一个有前景的方向。最大的5B T2I PanGu-Draw模型已在昇腾平台上发布。项目页面:https://pangu-draw.github.io

英文摘要:

Current large-scale diffusion models represent a giant leap forward in conditional image synthesis, capable of interpreting diverse cues like text, human poses, and edges. However, their reliance on substantial computational resources and extensive data collection remains a bottleneck. On the other hand, the integration of existing diffusion models, each specialized for different controls and operating in unique latent spaces, poses a challenge due to incompatible image resolutions and latent space embedding structures, hindering their joint use. Addressing these constraints, we present "PanGu-Draw", a novel latent diffusion model designed for resource-efficient text-to-image synthesis that adeptly accommodates multiple control signals. We first propose a resource-efficient Time-Decoupling Training Strategy, which splits the monolithic text-to-image model into structure and texture generators. Each generator is trained using a regimen that maximizes data utilization and computational efficiency, cutting data preparation by 48% and reducing training resources by 51%. Secondly, we introduce "Coop-Diffusion", an algorithm that enables the cooperative use of various pre-trained diffusion models with different latent spaces and predefined resolutions within a unified denoising process. This allows for multi-control image synthesis at arbitrary resolutions without the necessity for additional data or retraining. Empirical validations of Pangu-Draw show its exceptional prowess in text-to-image and multi-control image generation, suggesting a promising direction for future model training efficiencies and generation versatility. The largest 5B T2I PanGu-Draw model is released on the Ascend platform. Project page: $\href{https://pangu-draw.github.io}{this~https~URL}$

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑