面向文本到3D室内场景生成的视觉-语言模型中的全局-局部蒙特卡洛树搜索
Global-Local Monte Carlo Tree Search in Vision-Language Models for Text-to-3D Indoor Scene Generation
AI总结：
提出一种全局-局部蒙特卡洛树搜索方法，通过分层场景表示和PRM引导的MCTS解决文本到3D室内场景生成中的错误传播问题，并构建新基准数据集3DTindo-bench。
摘要：
大型视觉-语言模型在各种任务中取得了显著的推理性能。然而，关于使用LVLM进行文本到3D室内场景生成的研究很少。主要挑战在于，现有的基于LVLM的方法采用思维链顺序决策机制，无法修正早期决策，导致错误传播。在本文中，我们将该任务视为一个受空间和布局常识约束的规划问题。为解决此问题，我们将其建模为具有全局树和局部树的树搜索问题，这与现有的顺序决策方法不同。在全局树中，我们迭代地放置每个对象，并像人类布置房间一样探索多种尝试，其中问题空间表示为树。为了有效搜索树，我们提出了一种分层场景表示和PRM引导的MCTS方法。分层表示将场景抽象为房间级别、区域级别、地板对象级别和支撑对象级别。PRM引导的MCTS方法使用PRM剪枝不必要的分支，并使用MCTS算法平衡探索和利用，以更少的尝试获得最优解。在局部树中，它进一步将每个对象的放置分解为更细的子步骤，包括具体的放置参数。为了使场景整体外观一致，我们利用预训练的扩散图像生成模型为场景中的所有对象预测纹理。由于现有的文本到3D室内场景生成基准在规模和多样性上仍然有限，我们收集了一个新的大规模多样化数据集，包含65种场景类型和3,250条指令，具有不同的尺寸、布局和风格，命名为3DTindo-bench，以更好地评估最先进模型的能力。我们的实验表明，我们的方法比最先进的方法生成更逼真的3D场景。
英文摘要：
Large Vision-Language Models have achieved significant reasoning performance in various tasks. However, there are few studies on text-to-3D indoor scene generation with LVLMs. The main challenge is that prevailing LVLM-based methods employ chain-of-thought sequential decision mechanisms that cannot revise earlier decisions, causing error propagation. In this paper, we consider the task as a planning problem constrained by spatial and layout commonsense. To solve this problem, we model it as a tree search problem with global and local trees, which differs from existing sequential decision-making approaches. In the global tree, we place each object iteratively and explore multiple attempts like humans furnishing a room, where the problem space is represented as a tree. To effectively search the tree, we propose a hierarchical scene representation and a PRM-guided MCTS method. This representation abstracts a scene into room level, region level, floor object level, and supported object level. The PRM-guided MCTS method uses the PRM to prune unnecessary branches and the MCTS algorithm to balance exploration and exploitation to get an optimal solution with fewer attempts. In the local tree, it further decomposes the placement of each object into finer sub-steps, including the specific placement parameters. To make the whole appearance of the scene consistent, we leverage pre-trained diffusion image generative models to predict textures for all the objects in the scene. As existing benchmarks for text-to-3D indoor scene generation remain limited in scale and diversity, we collect a new large-scale diverse dataset that contains 65 scene types and 3250 instructions with diverse sizes, layouts, and styles, named 3DTindo-bench, to better assess the capability of the state-of-the-art models. Our experiments show that our method generates more realistic 3D scenes than state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
