一种用于自动驾驶中视觉语言模型幻觉缓解的低秩方法
A Low-Rank Method for Vision Language Model Hallucination Mitigation in Autonomous Driving
AI总结：
本文提出一种无需外部参考和模型访问的低秩方法，通过分解句子嵌入矩阵的稀疏残差对多VLM候选描述进行幻觉排序，在NuScenes上实现87%选择准确率并显著降低推理时间。
摘要：
视觉语言模型（VLMs）越来越多地被用于自动驾驶中以帮助理解交通场景，但它们有时会产生幻觉，即未基于视觉输入的错误细节。当地面真值参考不可用且模型内部不可访问时，检测和缓解幻觉具有挑战性。本文提出了一种新颖的自包含低秩方法，仅利用多个VLM生成的多条候选描述本身，无需外部参考或模型访问，即可根据幻觉水平自动对候选描述进行排序。通过构建句子嵌入矩阵并将其分解为低秩共识分量和稀疏残差，我们利用残差幅值对描述进行排序：选择残差最小的描述作为最无幻觉的描述。在NuScenes数据集上的实验表明，我们的方法在识别无幻觉描述方面达到了87%的选择准确率，相比未过滤基线提高了19%，相比多智能体辩论方法提高了6-10%。由稀疏误差幅值产生的排序与人类对幻觉的判断表现出强相关性，验证了我们的评分机制。此外，我们的方法易于并行化，与辩论方法相比将推理时间减少了51-67%，使其适用于实时自动驾驶应用。
英文摘要：
Vision Language Models (VLMs) are increasingly used in autonomous driving to help understand traffic scenes, but they sometimes produce hallucinations, which are false details not grounded in the visual input. Detecting and mitigating hallucinations is challenging when ground-truth references are unavailable and model internals are inaccessible. This paper proposes a novel self-contained low-rank approach to automatically rank multiple candidate captions generated by multiple VLMs based on their hallucination levels, using only the captions themselves without requiring external references or model access. By constructing a sentence-embedding matrix and decomposing it into a low-rank consensus component and a sparse residual, we use the residual magnitude to rank captions: selecting the one with the smallest residual as the most hallucination-free. Experiments on the NuScenes dataset demonstrate that our approach achieves 87% selection accuracy in identifying hallucination-free captions, representing a 19% improvement over the unfiltered baseline and a 6-10% improvement over multi-agent debate method. The sorting produced by sparse error magnitudes shows strong correlation with human judgments of hallucinations, validating our scoring mechanism. Additionally, our method, which can be easily parallelized, reduces inference time by 51-67% compared to debate approaches, making it practical for real-time autonomous driving applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
