MoVieDrive：基于多模态多视角视频扩散变换器的城市场景合成
MoVieDrive: Urban Scene Synthesis with Multi-Modal Multi-View Video Diffusion Transformer
AI总结：
本文提出MoVieDrive，通过多模态多视角视频扩散变换器生成城市驾驶场景视频，实现多模态数据生成与可控生成。
摘要：
城市场景合成与视频生成模型在自动驾驶中展现了巨大潜力。现有视频生成方法主要关注RGB视频生成，缺乏多模态视频生成能力。然而，深度图和语义图等多模态数据对自动驾驶中的整体城市场景理解至关重要。尽管使用多个模型生成不同模态是可行的，但这增加了模型部署的难度，并未利用多模态数据的互补信息。为了解决这个问题，本文提出了一种新的多模态多视角视频生成方法。具体而言，我们构建了一个由模态共享组件和模态特定组件组成的统一扩散变换器模型。然后，我们利用多样化的条件输入，将可控的场景结构和内容提示编码到多模态多视角统一扩散模型中。这样，我们的方法能够在统一框架中生成多模态多视角驾驶场景视频。在真实世界自动驾驶数据集上的彻底实验表明，我们的方法在视频生成质量和可控性方面优于最先进的方法，同时支持多模态多视角数据生成。
英文摘要：
Urban scene synthesis with video generation models has recently shown great potential for autonomous driving. Existing video generation approaches to autonomous driving primarily focus on RGB video generation and lack the ability to support multi-modal video generation. However, multi-modal data, such as depth maps and semantic maps, are crucial for holistic urban scene understanding in autonomous driving. Although it is feasible to use multiple models to generate different modalities, this increases the difficulty of model deployment and does not leverage complementary cues for multi-modal data generation. To address this problem, in this work, we propose a novel multi-modal multi-view video generation approach to autonomous driving. Specifically, we construct a unified diffusion transformer model composed of modal-shared components and modal-specific components. Then, we leverage diverse conditioning inputs to encode controllable scene structure and content cues into the multi-modal multi-view unified diffusion model. In this way, our approach is capable of generating multi-modal multi-view driving scene videos in a unified framework. Our thorough experiments on real-world autonomous driving dataset show that our approach achieves compelling video generation quality and controllability compared with state-of-the-art methods, while supporting multi-modal multi-view data generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
