arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.09748cs.CVcs.AI
收藏

基于结构感知扩散模型的参考图像与草图引导图像合成

Reference-based Image Composition with Sketch via Structure-aware Diffusion Model

KAIST(韩国科学技术院);Letsur Inc.(Letsur公司);Naver Webtoon(Naver网络漫画公司)

Kangyeol Kim, Sunghyun Park, Junsoo Lee, Jaegul Choo

AI总结:

本文提出结构感知扩散模型,联合参考图像与草图实现边缘级可控的局部图像编辑与补全,并通过实验验证其对任意场景进行用户驱动修改的能力。

摘要:

近期,大规模文本到图像生成模型取得显著进展,已在生成高保真图像方面展现出良好效果。为进一步增强可编辑性并实现细粒度生成,我们提出一种多输入条件图像合成模型,除参考图像外,还引入草图作为一种新的模态。借助草图提供的边缘级可控性,该方法使用户能够依据所需结构(即草图)和内容(即参考图像)编辑或补全图像的局部区域。我们的框架对预训练扩散模型进行微调,在保持草图引导的同时,利用参考图像补全缺失区域。尽管方法简单,却为满足用户获取所需图像的需求带来广泛机会。通过大量实验,我们证明所提方法为图像处理提供了独特用例,支持用户对任意场景进行自主修改。

英文摘要:

Recent remarkable improvements in large-scale text-to-image generative models have shown promising results in generating high-fidelity images. To further enhance editability and enable fine-grained generation, we introduce a multi-input-conditioned image composition model that incorporates a sketch as a novel modal, alongside a reference image. Thanks to the edge-level controllability using sketches, our method enables a user to edit or complete an image sub-part with a desired structure (i.e., sketch) and content (i.e., reference image). Our framework fine-tunes a pre-trained diffusion model to complete missing regions using the reference image while maintaining sketch guidance. Albeit simple, this leads to wide opportunities to fulfill user needs for obtaining the in-demand images. Through extensive experiments, we demonstrate that our proposed method offers unique use cases for image manipulation, enabling user-driven modifications of arbitrary scenes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑