arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.05591cs.CV
收藏

TweedieMix:改进基于扩散的图像/视频生成的多概念融合

TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generation

KRAFTON;Kim Jaechul Graduate School of AI, KAIST(KAIST金哉澈人工智能研究生院)

Gihyun Kwon, Jong Chul Ye

AI总结:

提出TweedieMix方法,通过两阶段采样策略在推理阶段组合定制扩散模型,有效融合多个性化概念,提升图像与视频生成保真度。

摘要:

尽管在定制文本到图像和视频生成模型方面取得了重大进展,但生成能有效整合多个个性化概念的图像和视频仍然是一项具有挑战性的任务。为了解决这个问题,我们提出了 TweedieMix,一种在推理阶段组合定制扩散模型的新方法。通过分析反向扩散采样的特性,我们的方法将采样过程分为两个阶段。在初始步骤中,我们应用多对象感知采样技术,以确保包含所需的目标对象。在后续步骤中,我们使用 Tweedie 公式在去噪图像空间中融合自定义概念的外观。我们的结果表明,与现有方法相比,TweedieMix 能够以更高的保真度生成多个个性化概念。此外,我们的框架可以轻松扩展到图像到视频扩散模型,从而能够生成包含多个个性化概念的视频。结果和源代码位于我们的匿名项目页面。

英文摘要:

Despite significant advancements in customizing text-to-image and video generation models, generating images and videos that effectively integrate multiple personalized concepts remains a challenging task. To address this, we present TweedieMix, a novel method for composing customized diffusion models during the inference phase. By analyzing the properties of reverse diffusion sampling, our approach divides the sampling process into two stages. During the initial steps, we apply a multiple object-aware sampling technique to ensure the inclusion of the desired target objects. In the later steps, we blend the appearances of the custom concepts in the de-noised image space using Tweedie's formula. Our results demonstrate that TweedieMix can generate multiple personalized concepts with higher fidelity than existing methods. Moreover, our framework can be effortlessly extended to image-to-video diffusion models, enabling the generation of videos that feature multiple personalized concepts. Results and source code are in our anonymous project page.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑