arXiv 2506.090422025-06-18cs.CV

Cosmos-Drive-Dreams：利用世界基础模型进行可扩展的合成驾驶数据生成

Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models

NVIDIA（英伟达）

Xuanchi Ren, Yifan Lu, Tianshi Cao, Ruiyuan Gao, Shengyu Huang, Amirmojtaba Sabour, Tianchang Shen, Tobias Pfaff, Jay Zhangjie Wu, Runjian Chen, Seung Wook Kim,… 展开作者

Xuanchi Ren, Yifan Lu, Tianshi Cao, Ruiyuan Gao, Shengyu Huang, Amirmojtaba Sabour, Tianchang Shen, Tobias Pfaff, Jay Zhangjie Wu, Runjian Chen, Seung Wook Kim, Jun Gao, Laura Leal-Taixe, Mike Chen, Sanja Fidler, Huan Ling

备注：: Only the core contributors are listed. The full list of contributors can be found in Appendix A of this paper

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Cosmos-Drive-Dreams合成数据生成流水线，基于Cosmos-Drive世界模型生成高保真驾驶视频，以缓解长尾分布并提升3D检测和驾驶策略学习的泛化能力。

摘要：

为安全关键的物理人工智能系统（如自动驾驶汽车（AV））收集和标注真实世界数据既耗时又昂贵。捕捉罕见边缘情况尤其具有挑战性，而这些情况在自动驾驶系统的训练和测试中起着关键作用。为应对这一挑战，我们引入了Cosmos-Drive-Dreams——一个合成数据生成（SDG）流水线，旨在生成具有挑战性的场景，以促进感知和驾驶策略训练等下游任务。为该流水线提供动力的是Cosmos-Drive，这是一套从NVIDIA Cosmos世界基础模型中专门针对驾驶领域优化的模型，能够生成可控、高保真、多视角且时空一致的驾驶视频。我们通过将Cosmos-Drive-Dreams应用于扩展驾驶数据集的数量和多样性，并包含高保真和具有挑战性的场景，展示了这些模型的实用性。实验上，我们证明生成的数据有助于缓解长尾分布问题，并增强下游任务（如3D车道检测、3D物体检测和驾驶策略学习）的泛化能力。我们通过NVIDIA的Cosmos平台开源了我们的流水线工具包、数据集和模型权重。项目页面：https://research.nvidia.com/labs/toronto-ai/cosmos_drive_dreams

英文摘要：

Collecting and annotating real-world data for safety-critical physical AI systems, such as Autonomous Vehicle (AV), is time-consuming and costly. It is especially challenging to capture rare edge cases, which play a critical role in training and testing of an AV system. To address this challenge, we introduce the Cosmos-Drive-Dreams - a synthetic data generation (SDG) pipeline that aims to generate challenging scenarios to facilitate downstream tasks such as perception and driving policy training. Powering this pipeline is Cosmos-Drive, a suite of models specialized from NVIDIA Cosmos world foundation model for the driving domain and are capable of controllable, high-fidelity, multi-view, and spatiotemporally consistent driving video generation. We showcase the utility of these models by applying Cosmos-Drive-Dreams to scale the quantity and diversity of driving datasets with high-fidelity and challenging scenarios. Experimentally, we demonstrate that our generated data helps in mitigating long-tail distribution problems and enhances generalization in downstream tasks such as 3D lane detection, 3D object detection and driving policy learning. We open source our pipeline toolkit, dataset and model weights through the NVIDIA's Cosmos platform. Project page: https://research.nvidia.com/labs/toronto-ai/cosmos_drive_dreams

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。