GaussianDWM: 基于3D高斯场景表示的统一场景理解和多模态生成驱动世界模型
GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generation
AI总结：
本文提出基于3D高斯表示的统一驱动世界模型框架，实现3D场景理解和多模态生成，并通过语言引导采样策略和双条件生成模型提升生成效果，实验验证其在nuScenes和NuInteract数据集上的优越性能。
摘要：
本文提出了一种基于3D高斯表示的统一驱动世界模型框架，旨在解决现有驱动世界模型（DWMs）在3D场景理解和多模态生成方面的不足。现有DWMs只能根据输入数据生成内容，无法解释或推理驾驶环境。此外，当前方法使用点云或BEV特征表示3D空间信息，无法准确对齐文本信息与底层3D场景。为了解决这些限制，我们提出了一种新的统一DWM框架，基于3D高斯场景表示，实现了3D场景理解和多模态生成，并能够为理解和生成任务提供上下文丰富性。我们的方法通过将丰富的语言特征嵌入到每个高斯体素中，直接对齐文本信息与3D场景，从而实现早期模态对齐。此外，我们设计了一种新颖的任务感知语言引导采样策略，以去除冗余的3D高斯体素，并将准确且紧凑的3D标记注入LLM中。此外，我们设计了一种双条件多模态生成模型，其中通过我们的视觉-语言模型捕获的信息作为高阶语言条件，与低阶图像条件结合，共同引导多模态生成过程。我们在nuScenes和NuInteract数据集上进行了全面研究，验证了我们框架的有效性。我们的方法实现了最先进的性能。我们将公开代码在GitHub上：https://github.com/dtc111111/GaussianDWM。
英文摘要：
Driving World Models (DWMs) have been developing rapidly with the advances of generative models. However, existing DWMs lack 3D scene understanding capabilities and can only generate content conditioned on input data, without the ability to interpret or reason about the driving environment. Moreover, current approaches represent 3D spatial information with point cloud or BEV features do not accurately align textual information with the underlying 3D scene. To address these limitations, we propose a novel unified DWM framework based on 3D Gaussian scene representation, which enables both 3D scene understanding and multi-modal scene generation, while also enabling contextual enrichment for understanding and generation tasks. Our approach directly aligns textual information with the 3D scene by embedding rich linguistic features into each Gaussian primitive, thereby achieving early modality alignment. In addition, we design a novel task-aware language-guided sampling strategy that removes redundant 3D Gaussians and injects accurate and compact 3D tokens into LLM. Furthermore, we design a dual-condition multi-modal generation model, where the information captured by our vision-language model is leveraged as a high-level language condition in combination with a low-level image condition, jointly guiding the multi-modal generation process. We conduct comprehensive studies on the nuScenes, and NuInteract datasets to validate the effectiveness of our framework. Our method achieves state-of-the-art performance. We will release the code publicly on GitHub https://github.com/dtc111111/GaussianDWM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
