SGEdit：连接LLM与Text2Image生成模型以实现基于场景图的图像编辑
SGEdit: Bridging LLM with Text2Image Generative Model for Scene Graph-based Image Editing
AI总结：
本文提出SGEdit框架，通过结合LLM与Text2Image扩散模型，利用场景图实现对象级精确修改与场景重组，显著提升了图像编辑精度和美学效果。
摘要：
场景图提供了一种结构化、层次化的图像表示，其中节点和边分别象征对象及其之间的关系。它可以作为图像编辑的自然接口，极大提高精度和灵活性。利用这一优势，我们引入了一个新框架，将大型语言模型（LLM）与Text2Image生成模型集成，用于基于场景图的图像编辑。这种集成实现了对象级别的精确修改和场景的创造性重组，同时不损害整体图像完整性。我们的方法包含两个主要阶段：1）利用LLM驱动的场景解析器，构建图像的场景图，捕获关键对象及其相互关系，并解析对象掩码和描述等细粒度属性。这些注释通过微调的扩散模型促进概念学习，用优化后的token和详细描述提示表示每个对象。2）在图像编辑阶段，LLM编辑控制器引导对特定区域的编辑。这些编辑随后由注意力调节的扩散编辑器实现，利用微调模型执行对象的添加、删除、替换和调整。通过大量实验，我们证明该框架在编辑精度和场景美学方面显著优于现有图像编辑方法。
英文摘要：
Scene graphs offer a structured, hierarchical representation of images, with nodes and edges symbolizing objects and the relationships among them. It can serve as a natural interface for image editing, dramatically improving precision and flexibility. Leveraging this benefit, we introduce a new framework that integrates large language model (LLM) with Text2Image generative model for scene graph-based image editing. This integration enables precise modifications at the object level and creative recomposition of scenes without compromising overall image integrity. Our approach involves two primary stages: 1) Utilizing a LLM-driven scene parser, we construct an image's scene graph, capturing key objects and their interrelationships, as well as parsing fine-grained attributes such as object masks and descriptions. These annotations facilitate concept learning with a fine-tuned diffusion model, representing each object with an optimized token and detailed description prompt. 2) During the image editing phase, a LLM editing controller guides the edits towards specific areas. These edits are then implemented by an attention-modulated diffusion editor, utilizing the fine-tuned model to perform object additions, deletions, replacements, and adjustments. Through extensive experiments, we demonstrate that our framework significantly outperforms existing image editing methods in terms of editing precision and scene aesthetics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
