arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.05463cs.CV
收藏

DriveScape:迈向高分辨率可控多视角驾驶视频生成

DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation

Tsinghua University(清华大学);Sensetime Research(商汤科技研究院);Northeastern University(东北大学);SenseAuto(毫末智行)

Wei Wu, Xi Guo, Weixuan Tang, Tingxuan Huang, Chiyu Wang, Dongyue Chen, Chenjing Ding

AI总结:

针对现有多视角驾驶视频生成方法时空一致性弱、帧率低的问题,提出端到端框架DriveScape,通过双向调制Transformer实现3D信息精准对齐,可生成10Hz高分辨率视频,在nuScenes数据集上取得SOTA结果。

摘要:

生成模型的最新进展为合成逼真驾驶视频提供了颇具前景的解决方案,这类视频对自动驾驶感知模型的训练至关重要。然而,现有方法在多视角视频生成方面往往存在不足,面临整合3D信息时难以维持时空一致性、难以通过统一模型有效学习的挑战。我们提出DriveScape,这是一个用于多视角、3D条件引导视频生成的端到端框架,能够以10Hz的频率生成1024×576分辨率的高分辨率视频。其他方法受3D框标注帧率限制仅能达到2Hz,DriveScape则凭借在稀疏条件下运行的能力突破了这一限制。我们提出的双向调制Transformer(BiMot)可确保3D结构信息的精准对齐,维持时空一致性。DriveScape在视频生成性能上表现优异,在nuScenes数据集上取得了当前最优结果,FID得分为8.34,FVD得分为76.39。我们的项目主页:https://metadrivescape.github.io/papers_project/drivescapev1/index.html

英文摘要:

Recent advancements in generative models have provided promising solutions for synthesizing realistic driving videos, which are crucial for training autonomous driving perception models. However, existing approaches often struggle with multi-view video generation due to the challenges of integrating 3D information while maintaining spatial-temporal consistency and effectively learning from a unified model. We propose DriveScape, an end-to-end framework for multi-view, 3D condition-guided video generation, capable of producing 1024 x 576 high-resolution videos at 10Hz. Unlike other methods limited to 2Hz due to the 3D box annotation frame rate, DriveScape overcomes this with its ability to operate under sparse conditions. Our Bi-Directional Modulated Transformer (BiMot) ensures precise alignment of 3D structural information, maintaining spatial-temporal consistency. DriveScape excels in video generation performance, achieving state-of-the-art results on the nuScenes dataset with an FID score of 8.34 and an FVD score of 76.39. Our project homepage: https://metadrivescape.github.io/papers_project/drivescapev1/index.html

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑