arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.05482cs.CVcs.AI
收藏

BARET:由目标文本反演驱动的基于平衡注意力的真实图像编辑

BARET : Balanced Attention based Real image Editing driven by Target-text Inversion

OPPO Research Institute(OPPO研究院);Tsinghua University(清华大学);Zhejiang University(浙江大学);Westlake University(西湖大学)

Yuming Qiao, Fanyi Wang, Jingwen Su, Yanhao Zhang, Yunjie Yu, Siyu Wu, Guo-Jun Qi

AI总结:

本文提出 BARET,通过目标文本反演、渐进过渡嵌入和平衡注意力模块,在不微调扩散模型的条件下实现多类型真实图像编辑,并以实验证明其优越效果与效率。

摘要:

基于扩散模型的图像编辑方法已得到快速发展,但其适用性受到特定编辑类型(例如前景或背景对象编辑、风格迁移)、多种条件(例如掩码、草图、描述文本)以及耗时的扩散模型微调等要求的限制。为缓解这些局限并实现高效的真实图像编辑,我们提出一种新颖的编辑技术,仅需输入图像和目标文本即可完成包括非刚性编辑在内的多种编辑类型,且无需微调扩散模型。我们的方法包含三项创新:(I)设计 Target-text Inversion Schedule(TTIS,目标文本反演调度),用于微调输入的目标文本嵌入,从而在没有图像描述文本的情况下实现快速图像重建并加速收敛。(II)Progressive Transition Scheme(渐进过渡方案)在目标文本嵌入与其微调版本之间应用渐进线性插值,以生成过渡嵌入,从而保持非刚性编辑能力。(III)Balanced Attention Module(BAM,平衡注意力模块)平衡文本描述与图像语义之间的权衡。通过结合重建过程中的自注意力图和过渡过程中的交叉注意力图,扩散过程中目标文本嵌入的引导得到优化。为证明所提出的 BARET 的编辑能力、有效性和效率,我们进行了大量定性与定量实验。此外,来自用户研究和消融研究的结果进一步证明了其相对于其他方法的优越性。

英文摘要:

Image editing approaches with diffusion models have been rapidly developed, yet their applicability are subject to requirements such as specific editing types (e.g., foreground or background object editing, style transfer), multiple conditions (e.g., mask, sketch, caption), and time consuming fine-tuning of diffusion models. For alleviating these limitations and realizing efficient real image editing, we propose a novel editing technique that only requires an input image and target text for various editing types including non-rigid edits without fine-tuning diffusion model. Our method contains three novelties:(I) Target-text Inversion Schedule (TTIS) is designed to fine-tune the input target text embedding to achieve fast image reconstruction without image caption and acceleration of convergence.(II) Progressive Transition Scheme applies progressive linear interpolation between target text embedding and its fine-tuned version to generate transition embedding for maintaining non-rigid editing capability.(III) Balanced Attention Module (BAM) balances the tradeoff between textual description and image semantics.By the means of combining self-attention map from reconstruction process and cross-attention map from transition process, the guidance of target text embeddings in diffusion process is optimized.In order to demonstrate editing capability, effectiveness and efficiency of the proposed BARET, we have conducted extensive qualitative and quantitative experiments. Moreover, results derived from user study and ablation study further prove the superiority over other methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑