MMGDreamer：用于几何可控3D室内场景生成的混合模态图
MMGDreamer: Mixed-Modality Graph for Geometry-Controllable 3D Indoor Scene Generation
AI总结：
提出MMGDreamer双分支扩散模型，利用混合模态图、视觉增强和关系预测实现几何可控的3D室内场景生成，达到最先进性能。
摘要：
可控的3D场景生成在虚拟现实和室内设计中具有广泛的应用，其中生成的场景应在几何方面表现出高度的真实感和可控性。场景图提供了一种适合这些应用的数据表示。然而，当前基于图的场景生成方法受限于基于文本的输入，并且对灵活的用户输入适应性不足，阻碍了精确控制对象几何的能力。为解决这一问题，我们提出了MMGDreamer，一种用于场景生成的双分支扩散模型，它融合了新颖的混合模态图、视觉增强模块和关系预测器。混合模态图允许对象节点整合文本和视觉模态，节点之间具有可选的关系。它增强了对灵活用户输入的适应性，并能够对生成场景中对象的几何进行精细控制。视觉增强模块通过使用文本嵌入构建视觉表示，丰富了纯文本节点的视觉保真度。此外，我们的关系预测器利用节点表示来推断节点之间缺失的关系，从而产生更连贯的场景布局。大量的实验结果表明，MMGDreamer在对象几何控制方面表现出优越性，达到了最先进的场景生成性能。项目页面：https://yangzhifeio.github.io/project/MMGDreamer。
英文摘要：
Controllable 3D scene generation has extensive applications in virtual reality and interior design, where the generated scenes should exhibit high levels of realism and controllability in terms of geometry. Scene graphs provide a suitable data representation that facilitates these applications. However, current graph-based methods for scene generation are constrained to text-based inputs and exhibit insufficient adaptability to flexible user inputs, hindering the ability to precisely control object geometry. To address this issue, we propose MMGDreamer, a dual-branch diffusion model for scene generation that incorporates a novel Mixed-Modality Graph, visual enhancement module, and relation predictor. The mixed-modality graph allows object nodes to integrate textual and visual modalities, with optional relationships between nodes. It enhances adaptability to flexible user inputs and enables meticulous control over the geometry of objects in the generated scenes. The visual enhancement module enriches the visual fidelity of text-only nodes by constructing visual representations using text embeddings. Furthermore, our relation predictor leverages node representations to infer absent relationships between nodes, resulting in more coherent scene layouts. Extensive experimental results demonstrate that MMGDreamer exhibits superior control of object geometry, achieving state-of-the-art scene generation performance. Project page: https://yangzhifeio.github.io/project/MMGDreamer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
