迈向通用且无需训练的文本引导语义操控
Towards Generalized and Training-Free Text-Guided Semantic Manipulation
AI总结：
本文提出一种通用且无需训练的文本引导语义操控方法GTF，利用扩散模型中噪声的几何特性实现添加、移除和风格迁移等多种编辑，支持即插即用和跨模态应用。
摘要：
文本引导的语义操控是指对由源提示生成的图像进行语义编辑，使其匹配目标提示，从而实现所需的语义变化（例如添加、移除和风格迁移），同时保留不相关的内容。凭借扩散模型强大的生成能力，该任务已展现出生成高保真视觉内容的潜力。然而，现有方法通常要么需要耗时的微调（效率低下），要么无法完成多种语义操控（扩展性差），并且/或者缺乏对不同模态任务的支持（通用性有限）。经过进一步研究，我们发现扩散模型中噪声的几何特性与语义变化密切相关。受此启发，我们提出了一种新颖的GTF方法用于文本引导的语义操控，该方法具有以下吸引人的能力：1）通用性：我们的GTF支持多种语义操控（例如添加、移除和风格迁移），并且可以无缝集成到所有基于扩散的方法中（即插即用），跨越不同模态（即模态无关）；2）无需训练：GTF通过简单地控制噪声之间的几何关系即可产生高保真结果，无需调优或优化。我们的大量实验证明了该方法的有效性，突显了其在推动语义操控领域最先进技术方面的潜力。
英文摘要：
Text-guided semantic manipulation refers to semantically editing an image generated from a source prompt to match a target prompt, enabling the desired semantic changes (e.g., addition, removal, and style transfer) while preserving irrelevant contents. With the powerful generative capabilities of the diffusion model, the task has shown the potential to generate high-fidelity visual content. Nevertheless, existing methods either typically require time-consuming fine-tuning (inefficient), fail to accomplish multiple semantic manipulations (poorly extensible), and/or lack support for different modality tasks (limited generalizability). Upon further investigation, we find that the geometric properties of noises in the diffusion model are strongly correlated with the semantic changes. Motivated by this, we propose a novel $\textit{GTF}$ for text-guided semantic manipulation, which has the following attractive capabilities: 1) $\textbf{Generalized}$: our $\textit{GTF}$ supports multiple semantic manipulations (e.g., addition, removal, and style transfer) and can be seamlessly integrated into all diffusion-based methods (i.e., Plug-and-play) across different modalities (i.e., modality-agnostic); and 2) $\textbf{Training-free}$: $\textit{GTF}$ produces high-fidelity results via simply controlling the geometric relationship between noises without tuning or optimization. Our extensive experiments demonstrate the efficacy of our approach, highlighting its potential to advance the state-of-the-art in semantics manipulation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
