ST-LDM：一种用于真实图像中文本引导对象生成的通用框架
ST-LDM: A Universal Framework for Text-Grounded Object Generation in Real Images
AI总结：
ST-LDM提出基于Swin-Transformer的通用框架，通过可变形多模态Transformer与全局感知自编码器，在无需训练的反向引导下提升扩散模型在文本引导对象生成中的空间定位能力。
摘要：
我们提出了一种新颖的图像编辑场景，称为文本引导对象生成（TOG），其定义为在真实图像中根据文本描述在空间上生成一个新对象。现有的扩散模型在复杂的现实世界场景中表现出空间感知的局限性，依赖额外的模态来施加约束，而TOG在语言信息的弱监督下对场景理解提出了更高的挑战。我们提出了一种基于Swin-Transformer的通用框架ST-LDM，它可以集成到任何潜在扩散模型中，并具有无需训练的反向引导。ST-LDM包含一个具有自适应压缩尺度和分层视觉特征的全局感知自编码器，并并行使用可变形多模态Transformer来为后续去噪过程生成区域级引导。我们通过引入可变形特征对齐来分层细化空间定位，并融合多尺度视觉和语言信息，从而超越了传统注意力机制仅关注现有视觉特征的局限。大量实验表明，我们的模型增强了注意力机制的定位能力，同时保留了扩散模型固有的生成能力。
英文摘要：
We present a novel image editing scenario termed Text-grounded Object Generation (TOG), defined as generating a new object in the real image spatially conditioned by textual descriptions. Existing diffusion models exhibit limitations of spatial perception in complex real-world scenes, relying on additional modalities to enforce constraints, and TOG imposes heightened challenges on scene comprehension under the weak supervision of linguistic information. We propose a universal framework ST-LDM based on Swin-Transformer, which can be integrated into any latent diffusion model with training-free backward guidance. ST-LDM encompasses a global-perceptual autoencoder with adaptable compression scales and hierarchical visual features, parallel with deformable multimodal transformer to generate region-wise guidance for the subsequent denoising process. We transcend the limitation of traditional attention mechanisms that only focus on existing visual features by introducing deformable feature alignment to hierarchically refine spatial positioning fused with multi-scale visual and linguistic information. Extensive Experiments demonstrate that our model enhances the localization of attention mechanisms while preserving the generative capabilities inherent to diffusion models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
