arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01991cs.CV
收藏

通过潜在向量预测实现扩散模型中的局部控制

Localized Control in Diffusion Models via Latent Vector Prediction

Pablo Domingo-Gregorio, Javier Ruiz-Hidalgo

AI总结:

本文提出通过潜在向量预测实现扩散模型的局部控制,使模型能精准控制用户定义区域并自主生成剩余部分。

摘要:

扩散模型在文本到图像生成中崭露头角,能够从文本描述中生成高质量图像。然而,仅通过文本实现详细控制以获得期望图像仍是一种繁琐的试错过程。最近的方法引入了图像级控制与文本提示相结合,利用先验图像提取边缘、分割和深度图等条件信息。尽管有效,这些方法在整张图像上统一应用条件,限制了局部控制。在本文中,我们提出了一种新的方法,以实现对图像上用户定义区域的精确局部控制,同时让扩散模型根据原始提示自主生成剩余区域。我们的方法引入了一种新的训练框架,结合掩码特征和额外的损失项,利用在任何扩散步骤中预测初始潜在向量来增强当前步骤与最终样本在潜在空间中的对应关系。广泛的实验表明,我们的方法有效合成高质量图像,具有受控的局部条件。

英文摘要:

Diffusion models emerged as a leading approach in text-to-image generation, producing high-quality images from textual descriptions. However, attempting to achieve detailed control to get a desired image solely through text remains a laborious trial-and-error endeavor. Recent methods have introduced image-level controls alongside with text prompts, using prior images to extract conditional information such as edges, segmentation and depth maps. While effective, these methods apply conditions uniformly across the entire image, limiting localized control. In this paper, we propose a novel methodology to enable precise local control over user-defined regions of an image, while leaving to the diffusion model the task of autonomously generating the remaining areas according to the original prompt. Our approach introduces a new training framework that incorporates masking features and an additional loss term, which leverages the prediction of the initial latent vector at any diffusion step to enhance the correspondence between the current step and the final sample in the latent space. Extensive experiments demonstrate that our method effectively synthesizes high-quality images with controlled local conditions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑