OmniDrive: 一种由LLM编排的多智能体世界模型,用于多视角驾驶视频生成的统一潜在协同压缩
OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation
AI总结:
提出DRIVE-CHOREO,一种由LLM编排的多智能体世界模型,通过三个Qwen2.5-VL智能体协同生成位置感知的潜在序列,并利用视图-时间置换与3D VAE协同压缩,实现可控多视角视频生成,在nuScenes上达到SOTA多视角一致性和BEV mAP 21.6。
摘要:
自动驾驶的生成式世界模型面临两个未解决的对立:异构控制注入(自由形式语言、高清地图、轨迹和相机位姿存在于不兼容的表示空间)和事后跨视图融合(每个相机的潜在编码未能编码全局3D几何)。我们将两者追溯到同一个根本原因:在潜在标记级别上缺乏对齐语言、几何和像素的共享符号中间语言。我们提出DRIVE-CHOREO,一种由LLM编排的多智能体世界模型,将可控多视图视频生成重新定义为潜在编排。三个Qwen2.5-VL智能体——一个解析用户意图为结构化WorldScript的导演,一个将其接地为空间锚定布局标记的制图师,以及一个将跨视图批评反馈为辅助监督的审计员——共同创作一个单一的位置感知标记序列。该序列通过视图-时间置换与多视图视频协同压缩,在3D VAE的卷积感受野内强制实现相机间几何。在nuScenes上,DRIVE-CHOREO以具有竞争力的FVD(45.7)实现了新的最先进的多视图一致性和BEV mAP(21.6);仅在我们的合成数据上训练的检测器在真实验证集上获得了+2.4 NDS,验证了下游实用性。
英文摘要:
Generative world models for autonomous driving face two unresolved tensions: heterogeneous control injection, where free-form language, HD-maps, trajectories, and camera poses reside in incompatible representational spaces, and post-hoc cross-view fusion, where per-camera latents fail to encode global 3-D geometry. We trace both to a single root cause: the absence of a shared symbolic interlingua aligning language, geometry, and pixels at the latent-token level. We present DRIVE-CHOREO, an LLM-choreographed multi-agent world model that recasts controllable multi-view video generation as latent choreography. Three Qwen2.5-VL agents - a Director parsing user intent into a structured WorldScript, a Cartographer grounding it into spatially-anchored layout tokens, and an Auditor feeding cross-view critiques back as auxiliary supervision - jointly author a single position-aware token sequence. This sequence is co-compressed with the multi-view video via a view-time permutation that enforces inter-camera geometry within the convolutional receptive field of a 3-D VAE. On nuScenes, DRIVE-CHOREO sets new state-of-the-art multi-view consistency and BEV mAP (21.6) with competitive FVD (45.7); a detector trained purely on our synthetic data gains +2.4 NDS on the real validation split, validating downstream utility.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。