TALE：基于自适应潜变量操控与能量引导优化的免训练跨域图像合成
TALE: Training-free Cross-domain Image Composition via Adaptive Latent Manipulation and Energy-guided Optimization
AI总结：
TALE提出免训练的跨域图像合成框架，通过在潜空间进行自适应操控和能量引导优化，解决注意力图方法难以保持物体身份和风格适配的问题，在真实与艺术领域均取得最先进性能。
摘要：
我们提出 TALE，一种新颖的免训练框架，利用文本到图像扩散模型的生成能力来解决跨域图像合成任务，该任务专注于将用户指定的物体无缝地融入指定的视觉场景中，而不受域差异的影响。以往的方法通常需要训练辅助网络或在定制数据集上微调扩散模型，这些方法成本高昂，并且可能削弱预训练扩散模型稳健的文本与视觉先验。近期一些工作试图通过提出免训练的变通方案来突破这一障碍，这些方案依赖于操控注意力图以隐式地约束去噪过程。然而，通过注意力图进行合成并不一定能产生期望的合成结果。这些方法通常只能保留部分语义信息，往往难以保持输入物体的身份特征，或在生成图像中表现出有限的背景与物体风格适配能力。相比之下，TALE 是一种直接在潜空间上操作的新方法，为合成过程提供显式且有效的引导以解决这些问题。具体而言，我们为 TALE 配备了两个机制，分别称为自适应潜变量操控（Adaptive Latent Manipulation）和能量引导潜变量优化（Energy-guided Latent Optimization）。前者通过直接利用对应时间步上的背景和前景潜变量，构建有利于启动和引导合成过程的含噪潜变量；后者利用指定的能量函数进一步优化中间潜变量，使其符合特定条件，从而补充前者以生成期望的最终结果。我们的实验表明，TALE 超越了先前基线，并在各种照片级真实和艺术领域的图像引导合成中取得了最先进的性能。
英文摘要：
We present TALE, a novel training-free framework harnessing the generative capabilities of text-to-image diffusion models to address the cross-domain image composition task that focuses on flawlessly incorporating user-specified objects into a designated visual contexts regardless of domain disparity. Previous methods often involve either training auxiliary networks or finetuning diffusion models on customized datasets, which are expensive and may undermine the robust textual and visual priors of pre-trained diffusion models. Some recent works attempt to break the barrier by proposing training-free workarounds that rely on manipulating attention maps to tame the denoising process implicitly. However, composing via attention maps does not necessarily yield desired compositional outcomes. These approaches could only retain some semantic information and usually fall short in preserving identity characteristics of input objects or exhibit limited background-object style adaptation in generated images. In contrast, TALE is a novel method that operates directly on latent space to provide explicit and effective guidance for the composition process to resolve these problems. Specifically, we equip TALE with two mechanisms dubbed Adaptive Latent Manipulation and Energy-guided Latent Optimization. The former formulates noisy latents conducive to initiating and steering the composition process by directly leveraging background and foreground latents at corresponding timesteps, and the latter exploits designated energy functions to further optimize intermediate latents conforming to specific conditions that complement the former to generate desired final results. Our experiments demonstrate that TALE surpasses prior baselines and attains state-of-the-art performance in image-guided composition across various photorealistic and artistic domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
