arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.19645cs.CV
收藏

一次学习,随处编辑:扩散模型的视觉方向迁移

Learn Once, Edit Anywhere: Visual Direction Transfer for Diffusion Models

Virginia Tech(弗吉尼亚理工大学)

Yusuf Dalva, Hidir Yesiltepe, Pinar Yanardag

AI总结:

提出ViDiT框架,通过从图像编辑对中学习连续编辑方向,实现无需微调的零样本图像属性编辑,在保持输入保真度的同时实现精确、可扩展的属性控制。

摘要:

扩散模型的快速发展使得从文本提示生成高保真图像成为可能,但实现对特定属性的精确、解耦控制仍然是一个重大挑战。一个根本限制在于,图像之间的视觉差异通常比通过人工构建的文本描述所能捕捉到的更为描述性和微妙,而文本描述往往无法传达细粒度的语义细节。为了解决这个问题,我们引入了ViDiT(扩散模型的视觉方向迁移),这是一个通过直接从图像编辑对中捕捉潜在语义来扩展编辑词汇的框架。ViDiT通过从一小批“前后”示例中优化单个全局连续编辑方向来学习底层变换。该优化过程将视觉变化转移到扩散模型的条件空间中,从而允许进行文本难以描述的细节编辑。ViDiT遵循“一次学习”原则,完全消除了推理过程中模型微调或昂贵的每图像优化的需要。一旦学习完成,这些连续方向便实现了“随处编辑”能力,允许用户以零样本方式对任何图像应用高度解耦的操作,如面部特征、动物属性或艺术风格的变化,并对编辑强度进行精细控制。定量和定性评估表明,ViDiT在保持输入保真度的同时,实现了精确、可扩展的属性控制,优于现有的基于文本的编辑方法。

英文摘要:

The rapid advancement of diffusion models has enabled the generation of high-fidelity images from textual prompts, yet achieving precise, disentangled control over specific attributes remains a significant challenge. A fundamental limitation arises because visual differences between images are often far more descriptive and nuanced than what can be captured through human-crafted text descriptions, which frequently fail to convey fine-grained semantic details. To address this, we introduce ViDiT (Visual Direction Transfer for Diffusion), a framework that expands the editing vocabulary by capturing latent semantics directly from image-edit pairs. ViDiT learns the underlying transformation by optimizing a single, global, and continuous editing direction from a small set of ``before-and-after'' examples. This optimization process transfers visual changes into the diffusion model's conditioning space, allowing for detailed edits that text alone cannot easily describe. ViDiT operates on a ``Learn Once'' principle, which completely eliminates the need for model fine-tuning or expensive per-image optimization during inference. Once learned, these continuous directions enable ``Edit Anywhere'' capabilities, allowing users to apply highly disentangled manipulations, such as changes in facial features, animal attributes, or artistic styles, to any image in a zero-shot manner with granular control over the edit intensity. Quantitative and qualitative evaluations demonstrate that ViDiT outperforms existing text-based editing methods in maintaining input faithfulness while achieving precise, scalable attribute control.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑