HoloDrive：面向自动驾驶的整体 2D-3D 多模态街景生成
HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving
AI总结：
本文提出 HoloDrive 框架，通过 BEV 与相机双向转换、深度预测分支及时间渐进训练，联合生成自动驾驶相机图像与 LiDAR 点云，并在多项基准上显著优于 SOTA。
摘要：
生成模型已显著提升了自动驾驶中相机图像或 LiDAR 点云的生成与预测质量。然而，真实世界的自动驾驶系统使用多种输入模态，通常是相机和 LiDAR，它们包含用于生成的互补信息，而现有生成方法忽略了这一关键特征，导致生成结果仅涵盖彼此分离的 2D 或 3D 信息。为填补自动驾驶中 2D-3D 多模态联合生成的空白，本文提出 HoloDrive 框架，用于联合生成相机图像和 LiDAR 点云。我们在异构生成模型之间采用 BEV-to-Camera 和 Camera-to-BEV 转换模块，并在 2D 生成模型中引入深度预测分支，以消除从图像空间反投影到 BEV 空间时的歧义；随后通过加入时间结构和精心设计的渐进式训练，将该方法扩展到未来预测。此外，我们在单帧生成和世界模型基准上进行实验，结果表明，在生成指标方面，本方法相较 SOTA 方法带来了显著性能提升。
英文摘要：
Generative models have significantly improved the generation and prediction quality on either camera images or LiDAR point clouds for autonomous driving. However, a real-world autonomous driving system uses multiple kinds of input modality, usually cameras and LiDARs, where they contain complementary information for generation, while existing generation methods ignore this crucial feature, resulting in the generated results only covering separate 2D or 3D information. In order to fill the gap in 2D-3D multi-modal joint generation for autonomous driving, in this paper, we propose our framework, \emph{HoloDrive}, to jointly generate the camera images and LiDAR point clouds. We employ BEV-to-Camera and Camera-to-BEV transform modules between heterogeneous generative models, and introduce a depth prediction branch in the 2D generative model to disambiguate the un-projecting from image space to BEV space, then extend the method to predict the future by adding temporal structure and carefully designed progressive training. Further, we conduct experiments on single frame generation and world model benchmarks, and demonstrate our method leads to significant performance gains over SOTA methods in terms of generation metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
