全局-局部树搜索在VLMs中用于3D室内场景生成
Global-Local Tree Search in VLMs for 3D Indoor Scene Generation
AI总结：
提出全局-局部树搜索算法，利用大型视觉语言模型将3D室内场景生成视为规划问题，通过分层分解和表情符号网格定位，生成更合理的场景。
摘要：
大型视觉语言模型（VLMs），如GPT-4，已在各个领域取得了显著成功。然而，关于使用VLMs进行3D室内场景生成的研究却很少。本文将这一任务视为一个受空间和布局常识约束的规划问题。为了用VLM解决该问题，我们提出了一种新的全局-局部树搜索算法。在全局层面，该方法依次放置每个对象，并在每次放置过程中探索多种放置方案，其中问题空间被表示为树。为了减少树的深度，我们分层分解场景结构，即房间级别、区域级别、地板对象级别和支撑对象级别。该算法独立生成不同区域中的地板对象以及放置在不同地板对象上的支撑对象。在局部层面，我们也分解子任务，即每个对象的放置，为多个步骤。算法搜索问题空间的树。为了利用VLM模型生成对象的位置，我们将俯视图空间离散化为密集网格，并用多样的表情符号填充每个单元格，使单元格彼此区分。我们向VLM提供表情符号网格，VLM通过描述表情符号的名称来生成对象的合理位置。定量和定性实验结果表明，我们的方法生成的3D场景比最先进的方法更合理。我们的源代码可在https://github.com/dw-dengwei/TreeSearchGen获取。
英文摘要：
Large Vision-Language Models (VLMs), such as GPT-4, have achieved remarkable success across various fields. However, there are few studies on 3D indoor scene generation with VLMs. This paper considers this task as a planning problem subject to spatial and layout common sense constraints. To solve the problem with a VLM, we propose a new global-local tree search algorithm. Globally, the method places each object sequentially and explores multiple placements during each placement process, where the problem space is represented as a tree. To reduce the depth of the tree, we decompose the scene structure hierarchically, i.e. room level, region level, floor object level, and supported object level. The algorithm independently generates the floor objects in different regions and supported objects placed on different floor objects. Locally, we also decompose the sub-task, the placement of each object, into multiple steps. The algorithm searches the tree of problem space. To leverage the VLM model to produce positions of objects, we discretize the top-down view space as a dense grid and fill each cell with diverse emojis to make to cells distinct. We prompt the VLM with the emoji grid and the VLM produces a reasonable location for the object by describing the position with the name of emojis. The quantitative and qualitative experimental results illustrate our approach generates more plausible 3D scenes than state-of-the-art approaches. Our source code is available at https://github.com/dw-dengwei/TreeSearchGen .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
