面向可泛化复杂图像生成的场景图解耦与组合
Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation
AI总结：
本文提出利用场景图解耦与组合实现可泛化复杂图像生成，通过SL-VAE、CMA和MLS提升生成多样性、可控性及图操控一致性。
摘要：
从自然语言或布局条件生成图像已取得令人振奋的进展。然而，由于对多个对象及其关系建模不足，这些方法难以忠实再现复杂场景。为解决该问题，我们利用场景图这一强大的结构化表示进行复杂图像生成。与以往直接使用场景图进行生成的工作不同，我们以可泛化的方式运用变分自编码器和扩散模型的生成能力，组合来自场景图的多样化解耦视觉线索。具体而言，我们首先提出一种语义-布局变分自编码器（SL-VAE），从输入场景图中联合推导（布局，语义），从而在一对多映射中实现更多样且合理的生成。随后，我们开发了一种与扩散模型集成的组合掩码注意力（CMA），将（布局，语义）与细粒度属性作为生成引导。为进一步在保持视觉内容一致的同时实现图操控，我们引入了一种多层采样器（MLS），以获得“隔离式”图像编辑效果。大量实验表明，我们的方法在生成合理性和可控性方面优于近期基于文本、布局或场景图的竞争方法。
英文摘要：
There has been exciting progress in generating images from natural language or layout conditions. However, these methods struggle to faithfully reproduce complex scenes due to the insufficient modeling of multiple objects and their relationships. To address this issue, we leverage the scene graph, a powerful structured representation, for complex image generation. Different from the previous works that directly use scene graphs for generation, we employ the generative capabilities of variational autoencoders and diffusion models in a generalizable manner, compositing diverse disentangled visual clues from scene graphs. Specifically, we first propose a Semantics-Layout Variational AutoEncoder (SL-VAE) to jointly derive (layouts, semantics) from the input scene graph, which allows a more diverse and reasonable generation in a one-to-many mapping. We then develop a Compositional Masked Attention (CMA) integrated with a diffusion model, incorporating (layouts, semantics) with fine-grained attributes as generation guidance. To further achieve graph manipulation while keeping the visual content consistent, we introduce a Multi-Layered Sampler (MLS) for an "isolated" image editing effect. Extensive experiments demonstrate that our method outperforms recent competitors based on text, layout, or scene graph, in terms of generation rationality and controllability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
