arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2505.20129cs.CVcs.GR

基于空间情境化视觉语言模型的智能体三维场景生成

Agentic 3D Scene Generation with Spatially Contextualized VLMs

HKUST(香港科技大学);Dartmouth College(达特茅斯学院)

Xinhang Liu, Yu-Wing Tai, Chi-Keung Tang

AI总结:

本文提出通过注入包含场景画像、语义点云和场景超图的空间情境,使视觉语言模型具备结构化三维场景生成、理解与编辑能力,并构建了支持资产生成、环境验证和路径规划的智能体三维场景生成流程。

摘要:

尽管视觉语言模型(VLMs)在多模态内容生成方面取得了最新进展,但其对结构化三维场景进行推理和生成的能力在很大程度上仍未得到充分探索。这一局限限制了它们在具身人工智能、沉浸式仿真和交互式三维应用等空间基础任务中的实用性。我们提出了一种新范式,通过注入持续演化的空间情境,使视觉语言模型能够生成、理解和编辑复杂的三维环境。该情境由多模态输入构建,包含三个组成部分:提供高层语义蓝图的场景画像、捕获物体级几何信息的语义标注点云,以及编码丰富空间关系的场景超图,其中包括一元、二元和高阶约束。这些组件共同为视觉语言模型提供了一个结构化的、具有几何感知能力的工作记忆,将其固有的多模态推理能力与结构化三维理解相结合,以实现有效的空间推理。在此基础上,我们开发了一个智能体三维场景生成流程,其中视觉语言模型迭代地读取和更新空间情境。该流程具备高质量资产生成与几何修复、带自动验证的环境搭建,以及由场景超图引导的人机工学调整等特性。实验表明,我们的框架能够处理多样且具有挑战性的输入,达到了先前工作所未见的泛化水平。进一步的结果证明,注入空间情境使视觉语言模型能够执行交互式场景编辑和路径规划等下游任务,表明其在计算机图形学、三维视觉和具身应用中的空间智能系统具有巨大潜力。项目页面:https://spatctxvlm.github.io/project_page/。

英文摘要:

Despite recent advances in multimodal content generation enabled by vision-language models (VLMs), their ability to reason about and generate structured 3D scenes remains largely underexplored. This limitation constrains their utility in spatially grounded tasks such as embodied AI, immersive simulations, and interactive 3D applications. We introduce a new paradigm that enables VLMs to generate, understand, and edit complex 3D environments by injecting a continually evolving spatial context. Constructed from multimodal input, this context consists of three components: a scene portrait that provides a high-level semantic blueprint, a semantically labeled point cloud capturing object-level geometry, and a scene hypergraph that encodes rich spatial relationships, including unary, binary, and higher-order constraints. Together, these components provide the VLM with a structured, geometry-aware working memory that integrates its inherent multimodal reasoning capabilities with structured 3D understanding for effective spatial reasoning. Building on this foundation, we develop an agentic 3D scene generation pipeline in which the VLM iteratively reads from and updates the spatial context. The pipeline features high-quality asset generation with geometric restoration, environment setup with automatic verification, and ergonomic adjustment guided by the scene hypergraph. Experiments show that our framework can handle diverse and challenging inputs, achieving a level of generalization not observed in prior work. Further results demonstrate that injecting spatial context enables VLMs to perform downstream tasks such as interactive scene editing and path planning, suggesting strong potential for spatially intelligent systems in computer graphics, 3D vision, and embodied applications. Project page: https://spatctxvlm.github.io/project_page/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑