arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.01223cs.CVcs.AIcs.GRcs.LG
收藏

特征泼溅:语言驱动的基于物理的场景合成与编辑

Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editing

University of California San Diego(加利福尼亚大学圣迭戈分校);Massachusetts Institute of Technology(麻省理工学院);Institute for Artificial Intelligence and Fundamental Interactions(人工智能与基本相互作用研究所)

Ri-Zhao Qiu, Ge Yang, Weijia Zeng, Xiaolong Wang

AI总结:

本文提出Feature Splatting,将视觉语言特征蒸馏到3D高斯中,通过文本查询实现半自动场景分解,并利用粒子模拟器自动赋予材料属性以合成基于物理的动态,统一了外观、几何、材质与语义表示。

摘要:

使用3D高斯基元进行场景表示,在建模静态和动态3D场景的外观方面已取得优异成果。然而,许多图形应用要求能够同时操纵对象的外观和物理属性。我们提出Feature Splatting,这是一种将基于物理的动态场景合成与来自视觉语言基础模型的丰富语义相统一的方法,并通过自然语言进行锚定。我们的第一个贡献是一种将高质量的、以对象为中心的视觉语言特征蒸馏到3D高斯中的方法,从而支持使用文本查询进行半自动场景分解。我们的第二个贡献是一种利用基于粒子的模拟器从原本静态的场景合成基于物理的动态的方法,其中材料属性通过文本查询自动分配。我们对这一流程中使用的关键技术进行了消融研究,以说明使用携带特征的3D高斯作为统一格式来表示外观、几何、材料属性和基于自然语言的语义所面临的挑战与机遇。项目网站:https://feature-splatting.github.io/

英文摘要:

Scene representations using 3D Gaussian primitives have produced excellent results in modeling the appearance of static and dynamic 3D scenes. Many graphics applications, however, demand the ability to manipulate both the appearance and the physical properties of objects. We introduce Feature Splatting, an approach that unifies physics-based dynamic scene synthesis with rich semantics from vision language foundation models that are grounded by natural language. Our first contribution is a way to distill high-quality, object-centric vision-language features into 3D Gaussians, that enables semi-automatic scene decomposition using text queries. Our second contribution is a way to synthesize physics-based dynamics from an otherwise static scene using a particle-based simulator, in which material properties are assigned automatically via text queries. We ablate key techniques used in this pipeline, to illustrate the challenge and opportunities in using feature-carrying 3D Gaussians as a unified format for appearance, geometry, material properties and semantics grounded on natural language. Project website: https://feature-splatting.github.io/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑