从片段到场景：自动驾驶中基于视觉语言模型的时间理解
From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Models
AI总结：
提出自动驾驶时间理解基准TAD，通过场景思维链和轨迹认知图两种无训练方法提升视觉语言模型的时间推理能力。
摘要：
视觉语言模型（VLM）越来越多地被部署为野外自主代理的感知和推理骨干，其中自动驾驶（AD）是最安全关键的实例之一。可靠的时间理解对于此类代理预测事件、归因原因和在动态环境中安全行动至关重要，但即使对于最先进的（SoTA）VLM来说，这仍然是一个重大挑战。先前的视频基准强调了其他内容（体育、烹饪等），但现有基准没有专门关注短时和长时AD视频的时间理解。为填补这一空白，我们提出了自动驾驶时间理解（TAD）基准，包含近6000个问答（QA）对，涵盖7个任务，并评估了9个闭源和开源通用以及AD专用模型。当前SoTA模型在TAD上的表现远低于人类准确率。为了改进基于VLM的驾驶代理的时间推理，我们提出了两种新颖的无训练解决方案：Scene-CoT，它使用思维链（CoT）推理；以及TCogMap，它结合了由轨迹分析模块生成的自我中心时间认知图，该模块作为VLM周围的代理工具运行。与现有VLM集成后，我们的方法在TAD上的平均准确率提高了高达17.72%，在STSBench上提高了高达10.35%。通过引入TAD、对SoTA模型进行基准测试并提出有效的增强方法，本工作旨在促进野外代理AD系统时间理解的进一步进展。基准和评估代码分别可在${\href{https://huggingface.co/datasets/vbdai/TAD}{ ext{Hugging Face}}}$和${\href{https://github.com/vbdi/tad_bench}{ ext{GitHub}}}$上获取。
英文摘要：
Vision-Language Models (VLMs) are increasingly deployed as the perception and reasoning backbone of autonomous agents acting in the wild, with autonomous driving (AD) being one of the most safety-critical instances. Reliable temporal understanding is essential for such agents to anticipate events, attribute causes, and act safely in dynamic environments, yet this remains a significant challenge even for state-of-the-art (SoTA) VLMs. Prior video benchmarks have emphasized other content (sports, cooking, etc.), yet no existing benchmark focuses exclusively on temporal understanding for both short- and long-form AD footage. To fill this gap, we present the Temporal Understanding in Autonomous Driving (TAD) benchmark, comprising nearly 6000 question-answer (QA) pairs across 7 tasks, and evaluate 9 closed- and open-source generalist as well as AD-specialist models. Current SoTA models perform substantially below human accuracy on TAD. To improve the temporal reasoning of VLM-based driving agents, we propose two novel training-free solutions: Scene-CoT, which uses Chain-of-Thought (CoT) reasoning, and TCogMap, which incorporates an ego-centric temporal cognitive map produced by a trajectory-analysis module that operates as an agentic tool around the VLM. Integrated with existing VLMs, our methods improve average accuracy on TAD by up to $17.72\%$ and by up to $10.35\%$ on STSBench. By introducing TAD, benchmarking SoTA models, and proposing effective enhancements, this work aims to catalyze further progress on temporal understanding for agentic AD systems operating in the wild. The benchmark and evaluation code are available at ${\href{https://huggingface.co/datasets/vbdai/TAD}{\text{Hugging Face}}}$ and ${\href{https://github.com/vbdi/tad_bench}{\text{GitHub}}}$, respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
