OmniDrive-R1: 基于强化学习的交错多模态链式推理用于可信的视觉-语言自动驾驶
OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Driving
AI总结：
本文提出OmniDrive-R1，通过交错多模态链式推理机制统一感知与推理，引入强化驱动的视觉 grounding 能力，提升自动驾驶中的推理准确性和稳定性，实验显示其在DriveLMM-o1数据集上的表现显著优于基线模型。
摘要：
视觉-语言模型（VLMs）在安全关键领域如自动驾驶（AD）中的部署受到可靠性故障的严重阻碍，尤其是物体幻觉问题。这种故障源于其依赖于未接地的文本基链式推理（CoT）。尽管现有多模态CoT方法试图缓解这一问题，但它们存在两个根本性缺陷：（1）感知和推理阶段解耦，阻碍了端到端联合优化；（2）依赖昂贵的密集定位标签。因此，我们引入OmniDrive-R1，一种为自动驾驶设计的端到端VLM框架，通过交错多模态链式推理（iMCoT）机制统一感知和推理。我们的核心创新是强化驱动的视觉 grounding 能力，使模型能够自主引导其注意力并“聚焦”于关键区域进行细粒度分析。这一能力由我们纯粹的两阶段强化学习训练管道和Clip-GRPO算法实现。关键的是，Clip-GRPO引入了一种无标注、基于过程的接地奖励。该奖励不仅消除了对密集标签的需求，还通过强制实时跨模态一致性，避免了外部工具调用的不稳定性。在DriveLMM-o1上的大量实验表明，我们的模型在性能上有显著提升。与基线Qwen2.5VL-7B相比，OmniDrive-R1将整体推理得分从51.77%提升到80.35%，最终答案准确率从37.81%提升到73.62%。
英文摘要：
The deployment of Vision-Language Models (VLMs) in safety-critical domains like autonomous driving (AD) is critically hindered by reliability failures, most notably object hallucination. This failure stems from their reliance on ungrounded, text-based Chain-of-Thought (CoT) reasoning. While existing multi-modal CoT approaches attempt mitigation, they suffer from two fundamental flaws: (1) decoupled perception and reasoning stages that prevent end-to-end joint optimization, and (2) reliance on expensive, dense localization labels. Thus we introduce OmniDrive-R1, an end-to-end VLM framework designed for autonomous driving, which unifies perception and reasoning through an interleaved Multi-modal Chain-of-Thought (iMCoT) mechanism. Our core innovation is an Reinforcement-driven visual grounding capability, enabling the model to autonomously direct its attention and "zoom in" on critical regions for fine-grained analysis. This capability is enabled by our pure two-stage reinforcement learning training pipeline and Clip-GRPO algorithm. Crucially, Clip-GRPO introduces an annotation-free, process-based grounding reward. This reward not only eliminates the need for dense labels but also circumvents the instability of external tool calls by enforcing real-time cross-modal consistency between the visual focus and the textual reasoning. Extensive experiments on DriveLMM-o1 demonstrate our model's significant improvements. Compared to the baseline Qwen2.5VL-7B, OmniDrive-R1 improves the overall reasoning score from 51.77% to 80.35%, and the final answer accuracy from 37.81% to 73.62%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
