SceneAssistant: 一种用于开放词汇3D场景生成的视觉反馈代理
SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generation
AI总结：
SceneAssistant通过视觉反馈驱动代理实现开放词汇3D场景生成，结合3D物体生成模型和视觉-语言模型的空间推理能力，生成高质量且多样化的3D场景。
摘要：
从自然语言生成3D场景对于数字内容创作具有高度的吸引力。然而，现有方法大多领域受限或依赖于预定义的空间关系，限制了其在无约束、开放词汇3D场景合成中的能力。在本文中，我们引入SceneAssistant，一种用于开放词汇3D场景生成的视觉反馈驱动代理。我们的框架利用现代3D物体生成模型以及视觉-语言模型（VLMs）的空间推理和规划能力。为了实现开放词汇场景组成，我们为VLMs提供了一套全面的原子操作（例如，缩放、旋转、聚焦）。在每次交互步骤中，VLM接收渲染的视觉反馈并相应地采取行动，通过迭代优化场景，以实现更一致的空间布局和更好的输入文本对齐。实验结果表明，我们的方法可以生成多样、开放词汇和高质量的3D场景。定性分析和定量人类评估均显示，我们的方法在现有方法上具有优越性。此外，我们的方法允许用户通过自然语言命令指示代理编辑现有场景。我们的代码可在https://github.com/ROUJINN/SceneAssistant上获得。
英文摘要：
Text-to-3D scene generation from natural language is highly desirable for digital content creation. However, existing methods are largely domain-restricted or reliant on predefined spatial relationships, limiting their capacity for unconstrained, open-vocabulary 3D scene synthesis. In this paper, we introduce SceneAssistant, a visual-feedback-driven agent designed for open-vocabulary 3D scene generation. Our framework leverages modern 3D object generation model along with the spatial reasoning and planning capabilities of Vision-Language Models (VLMs). To enable open-vocabulary scene composition, we provide the VLMs with a comprehensive set of atomic operations (e.g., Scale, Rotate, FocusOn). At each interaction step, the VLM receives rendered visual feedback and takes actions accordingly, iteratively refining the scene to achieve more coherent spatial arrangements and better alignment with the input text. Experimental results demonstrate that our method can generate diverse, open-vocabulary, and high-quality 3D scenes. Both qualitative analysis and quantitative human evaluations demonstrate the superiority of our approach over existing methods. Furthermore, our method allows users to instruct the agent to edit existing scenes based on natural language commands. Our code is available at https://github.com/ROUJINN/SceneAssistant
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
