RAC3：面向自动驾驶的基于检索增强的视觉语言模型边角案例理解方法
RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Models
AI总结：
针对视觉语言模型处理自动驾驶边角案例时存在幻觉、 grounding 不足的问题，提出检索增强框架RAC3，通过多模块设计与多模态思维链策略，在基准测试与端到端框架集成中取得更优的边角案例理解性能。
摘要：
理解并处理边角案例对于保障自动驾驶系统的安全性与可靠性至关重要。视觉语言模型（VLMs）在提升场景理解能力方面发挥着关键作用，但这类模型存在幻觉、现实场景 grounding 不足等显著缺陷，会削弱其在关键驾驶场景下的性能。本研究提出了新型框架RAC3，用于提升视觉语言模型的边角案例理解性能。RAC3集成了频空融合（FSF）图像编码器、结合难负例与半难负例挖掘的嵌入模型跨模态对齐训练方法，以及基于K-Means聚类与层次可导航小世界（HNSW）索引的快速查询检索流水线。研究还引入了多模态思维链（CoT）提示策略，用于在推理阶段引导类比推理、减少幻觉。此外，RAC3内置更新机制，可保障框架内的持续学习。在CODA与nuScenes数据集上开展的大量实验表明，RAC3在多项下游任务中显著提升了边角案例理解效果。与此前的最优方法相比，RAC3在CODA-LM基准上取得了74.46的最高得分，在与DriveLM等端到端框架集成时也展现出稳定的性能提升。上述结果证明，检索增强策略与跨模态对齐能够有效支撑更安全、可解释性更强的自动驾驶实现。
英文摘要：
Understanding and addressing corner cases is essential for ensuring the safety and reliability of autonomous driving systems. Vision-language models (VLMs) play a crucial role in enhancing scenario comprehension, yet they face significant challenges, such as hallucination and insufficient real-world grounding, which compromise their performance in critical driving scenarios. In this work, RAC3, a novel framework designed to enhance the performance of VLMs in corner case comprehension, is proposed. RAC3 integrates a frequency-spatial fusion (FSF) image encoder, a cross-modal alignment training method for embedding models with hard and semi-hard negative mining, and a fast querying and retrieval pipeline based on K-Means clustering and hierarchical navigable small world (HNSW) indexing. A multimodal chain-of-thought (CoT) prompting strategy to guide analogical reasoning and reduce hallucinations during inference is introduced. Moreover, an update mechanism is integrated into RAC3 to ensure continual learning within the framework. Extensive experiments on the CODA and nuScenes datasets demonstrate that RAC3 significantly improves corner case comprehension across multiple downstream tasks. Compared to prior state-of-the-art methods, RAC3 achieves the highest final score of 74.46 on the CODA-LM benchmark and shows consistent performance gains when integrated with end-to-end frameworks like DriveLM. These results demonstrate the effectiveness of retrieval-augmented strategies and cross-modal alignment for safer and more interpretable autonomous driving.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
