用于可控图像生成的空间感知潜变量初始化方法
Spatial-Aware Latent Initialization for Controllable Image Generation
AI总结：
针对文本到图像扩散模型布局控制效果差的问题，提出基于空间感知初始化噪声的即插即用方法，可集成到免训练布局框架中，在Stable Diffusion和COCO上验证了其提升布局引导效果且保留生成质量的作用。
摘要：
近年来，文本到图像扩散模型已展现出根据文本输入生成高质量图像的出色能力，但这类模型难以精准遵循文本指令中的空间布局信息。现有研究主要聚焦于让交叉注意力图与布局条件对齐，却忽略了初始化噪声对布局引导的影响。为实现更优的布局控制，我们提出在去噪过程中利用空间感知初始化噪声。具体而言，我们发现经过有限反演步骤的参考图像反演结果包含关于物体位置的宝贵空间感知信息，能让生成图像呈现相似布局。基于这一观察，我们构建了一个开放词汇框架，可为每个布局条件定制空间感知初始化噪声。除初始化噪声外无需修改其他模块，我们的方法可作为即插即用模块无缝集成到其他免训练布局引导框架中。我们在Stable Diffusion模型和COCO数据集上开展了定量与定性评估。搭载空间感知潜变量初始化后，我们的方法在保持高质量生成内容的同时，显著提升了布局引导的有效性。
英文摘要：
Recently, text-to-image diffusion models have demonstrated impressive ability to generate high-quality images conditioned on the textual input. However, these models struggle to accurately adhere to textual instructions regarding spatial layout information. While previous research has primarily focused on aligning cross-attention maps with layout conditions, they overlook the impact of the initialization noise on the layout guidance. To achieve better layout control, we propose leveraging a spatial-aware initialization noise during the denoising process. Specifically, we find that the inverted reference image with finite inversion steps contains valuable spatial awareness regarding the object's position, resulting in similar layouts in the generated images. Based on this observation, we develop an open-vocabulary framework to customize a spatial-aware initialization noise for each layout condition. Without modifying other modules except the initialization noise, our approach can be seamlessly integrated as a plug-and-play module within other training-free layout guidance frameworks. We evaluate our approach quantitatively and qualitatively on the available Stable Diffusion model and COCO dataset. Equipped with the spatial-aware latent initialization, our method significantly improves the effectiveness of layout guidance while preserving high-quality content.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
