Vega：通过自然语言指令学习驾驶
Vega: Learning to Drive with Natural Language Instructions
AI总结：
本文提出Vega模型，通过自然语言指令生成和规划，提升自动驾驶的灵活性和个性化水平，基于大规模驾驶数据集InstructScene进行实验验证。
摘要：
视觉-语言-世界-动作模型已重塑自动驾驶，使其能将语言纳入决策过程。然而，现有方法多仅利用语言模态进行场景描述或推理，缺乏灵活遵循多样化用户指令的能力。为此，我们构建了一个包含约10万场景的大规模驾驶数据集（InstructScene），每个场景均标注有多种驾驶指令及对应轨迹。我们提出统一的视觉-语言-世界-动作模型Vega，用于基于指令的生成和规划。我们采用自回归范式处理视觉输入（视觉）和语言指令（语言），并采用扩散范式生成未来预测（世界建模）和轨迹（动作）。我们进行联合注意力以促进模态间的交互，并使用不同的投影层以提升不同模态的能力。大量实验表明，我们的方法不仅在规划性能上表现优异，还展现出强大的指令遵循能力，为更智能和个性化的驾驶系统铺平了道路。
英文摘要：
Vision-language-action models have reshaped autonomous driving to incorporate languages into the decision-making process. However, most existing pipelines only utilize the language modality for scene descriptions or reasoning and lack the flexibility to follow diverse user instructions for personalized driving. To address this, we first construct a large-scale driving dataset (InstructScene) containing around 100,000 scenes annotated with diverse driving instructions with the corresponding trajectories. We then propose a unified Vision-Language-World-Action model, Vega, for instruction-based generation and planning. We employ the autoregressive paradigm to process visual inputs (vision) and language instructions (language) and the diffusion paradigm to generate future predictions (world modeling) and trajectories (action). We perform joint attention to enable interactions between the modalities and use individual projection layers for different modalities for more capabilities. Extensive experiments demonstrate that our method not only achieves superior planning performance but also exhibits strong instruction-following abilities, paving the way for more intelligent and personalized driving systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
