扩散Transformer中潜在空间的解耦实现零样本细粒度语义编辑
Latent Space Disentanglement in Diffusion Transformers Enables Zero-shot Fine-grained Semantic Editing
AI总结：
本文揭示了扩散Transformer潜在空间的解耦特性，并提出EMS框架实现零样本细粒度图像编辑，同时引入新指标和综合基准评估编辑性能。
摘要：
扩散Transformer（DiTs）在多样且高质量的文本到图像（T2I）生成中取得了显著成功。然而，文本和图像潜在空间如何单独及共同影响生成图像的语义，仍有待深入研究。通过对DiT潜在空间的调查，我们发现了能够实现零样本细粒度语义编辑的关键结论：（1）DiTs中的文本和图像空间本身是可分解的。（2）这些空间共同构成了一个解耦的语义表示空间，能够实现精确且细粒度的语义控制。（3）有效的图像编辑需要结合使用文本和图像潜在空间。基于这些见解，我们提出了一个简单有效的提取-操作-采样（EMS）框架用于零样本细粒度图像编辑。该方法首先利用多模态大语言模型将输入图像和编辑目标转换为文本描述。随后，我们根据期望的编辑程度线性操作文本嵌入，并采用约束分数蒸馏采样来操作图像嵌入。我们通过提出一种新指标来量化扩散模型潜在空间的解耦程度。为评估细粒度编辑性能，我们引入了一个包含人工标注、人工评估和自动指标的综合性基准。我们进行了广泛的实验和深入分析，以全面揭示扩散Transformer的语义解耦特性及所提方法的有效性。我们的标注基准数据集已公开，以促进该领域的可复现研究。
英文摘要：
Diffusion Transformers (DiTs) have achieved remarkable success in diverse and high-quality text-to-image(T2I) generation. However, how text and image latents individually and jointly contribute to the semantics of generated images, remain largely unexplored. Through our investigation of DiT's latent space, we have uncovered key findings that unlock the potential for zero-shot fine-grained semantic editing: (1) Both the text and image spaces in DiTs are inherently decomposable. (2) These spaces collectively form a disentangled semantic representation space, enabling precise and fine-grained semantic control. (3) Effective image editing requires the combined use of both text and image latent spaces. Leveraging these insights, we propose a simple and effective Extract-Manipulate-Sample (EMS) framework for zero-shot fine-grained image editing. Our approach first utilizes a multi-modal Large Language Model to convert input images and editing targets into text descriptions. We then linearly manipulate text embeddings based on the desired editing degree and employ constrained score distillation sampling to manipulate image embeddings. We quantify the disentanglement degree of the latent space of diffusion models by proposing a new metric. To evaluate fine-grained editing performance, we introduce a comprehensive benchmark incorporating both human annotations, manual evaluation, and automatic metrics. We have conducted extensive experimental results and in-depth analysis to thoroughly uncover the semantic disentanglement properties of the diffusion transformer, as well as the effectiveness of our proposed method. Our annotated benchmark dataset is publicly available at https://anonymous.com/anonymous/EMS-Benchmark, facilitating reproducible research in this domain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
