基于扩散模型的接地场景图与图像联合生成建模
Joint Generative Modeling of Grounded Scene Graphs and Images via Diffusion Models
AI总结：
该研究提出基于扩散模型的DiffuseSG框架，通过分解策略先生成异质属性的接地场景图再生成图像，在多个数据集和下游任务中表现优于现有方法。
摘要：
我们提出了一种用于接地场景图-图像联合生成的框架，这是一项涉及高维多模态结构化数据的挑战性任务。为了有效建模这种复杂的联合分布，我们采用了一种分解方法：首先生成接地场景图，随后基于生成的接地场景图进行图像生成。尽管条件图像生成已在文献中得到广泛探索，但我们的主要关注点是从噪声中生成接地场景图，这为图像生成过程提供了高效且可解释的控制。该任务需要生成具有合理属性的接地场景图，其节点（物体）和边（物体间关系）均具备异质属性，涵盖连续属性（例如物体边界框）和离散属性（例如物体与关系类别）。为应对这一挑战，我们提出了DiffuseSG——一种新型扩散模型，可联合建模异质的节点与边属性。我们探索了不同的编码策略以有效处理分类数据。DiffuseSG利用图Transformer作为去噪器，在连续空间中逐步细化接地场景图表示，随后将其离散化以生成结构化输出。此外，我们引入了一项基于交并比（IoU）的正则化项以提升实证性能。我们的模型在VG和COCO-Stuff数据集上的接地场景图生成任务中优于现有方法，在标准指标以及我们新提出的、能更准确捕捉任务复杂性的指标上均表现出色。此外，我们还展示了DiffuseSG在两项重要下游任务中的更广泛适用性：1）在一系列接地场景图补全任务中取得了优异结果；2）通过利用DiffuseSG生成的额外训练样本，提升了接地场景图检测模型的性能。
英文摘要：
We introduce a framework for joint grounded scene graph - image generation, a challenging task involving high-dimensional, multi-modal structured data. To effectively model this complex joint distribution, we adopt a factorized approach: first generating a grounded scene graph, followed by image generation conditioned on the generated grounded scene graph. While conditional image generation has been widely explored in the literature, our primary focus is on the generation of grounded scene graphs from noise, which provides efficient and interpretable control over the image generation process. This task requires generating plausible grounded scene graphs with heterogeneous attributes for both nodes (objects) and edges (relations among objects), encompassing continuous attributes (e.g., object bounding boxes) and discrete attributes (e.g., object and relation categories). To address this challenge, we introduce DiffuseSG, a novel diffusion model that jointly models the heterogeneous node and edge attributes. We explore different encoding strategies to effectively handle the categorical data. Leveraging a graph transformer as the denoiser, DiffuseSG progressively refines grounded scene graph representations in a continuous space before discretizing them to generate structured outputs. Additionally, we introduce an IoU-based regularization term to enhance empirical performance. Our model outperforms existing methods in grounded scene graph generation on the VG and COCO-Stuff datasets, excelling in both standard and newly introduced metrics that more accurately capture the task's complexity. Furthermore, we demonstrate the broader applicability of DiffuseSG in two important downstream tasks: 1) achieving superior results in a range of grounded scene graph completion tasks, and 2) enhancing grounded scene graph detection models by leveraging additional training samples generated by DiffuseSG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
