arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.06070cs.CV
收藏

ControlNeXt:面向图像与视频生成的强大且高效的控制方法

ControlNeXt: Powerful and Efficient Control for Image and Video Generation

CUHK(香港中文大学);SmartMore

Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Ming-Chang Yang, Jiaya Jia

AI总结:

ControlNeXt 提出一种轻量高效的可控图像与视频生成架构,以极小额外成本替代繁重控制分支,可无缝集成 LoRA 实现风格变换,并通过 Cross Normalization 实现快速稳定训练,同时减少多达 90% 的可学习参数。

摘要:

扩散模型在图像和视频生成中已展现出卓越且稳健的能力。为了对生成结果实现更强的控制,研究者引入了额外架构,如 ControlNet、Adapters 和 ReferenceNet,以整合条件控制。然而,当前的可控生成方法通常需要大量额外计算资源,尤其是在视频生成中,并且在训练上面临挑战或控制能力较弱。本文提出 ControlNeXt:一种用于可控图像和视频生成的强大且高效的方法。我们首先设计了一种更简洁高效的架构,以相对于基础模型而言极小的额外成本替代繁重的附加分支。这种简洁的结构还使我们的方法能够与其它 LoRA 权重无缝集成,从而无需额外训练即可实现风格变换。在训练方面,与其它替代方案相比,我们减少了多达 90% 的可学习参数。此外,我们提出另一种名为 Cross Normalization(CN)的方法,用于替代 Zero-Convolution,以实现快速且稳定的训练收敛。我们在图像和视频上使用不同基础模型进行了多种实验,证明了该方法的稳健性。

英文摘要:

Diffusion models have demonstrated remarkable and robust abilities in both image and video generation. To achieve greater control over generated results, researchers introduce additional architectures, such as ControlNet, Adapters and ReferenceNet, to integrate conditioning controls. However, current controllable generation methods often require substantial additional computational resources, especially for video generation, and face challenges in training or exhibit weak control. In this paper, we propose ControlNeXt: a powerful and efficient method for controllable image and video generation. We first design a more straightforward and efficient architecture, replacing heavy additional branches with minimal additional cost compared to the base model. Such a concise structure also allows our method to seamlessly integrate with other LoRA weights, enabling style alteration without the need for additional training. As for training, we reduce up to 90% of learnable parameters compared to the alternatives. Furthermore, we propose another method called Cross Normalization (CN) as a replacement for Zero-Convolution' to achieve fast and stable training convergence. We have conducted various experiments with different base models across images and videos, demonstrating the robustness of our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑