arXiv 2503.193552025-03-26cs.CV

ST-VLM：面向视觉语言模型时空推理的运动学指令微调

ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in Vision-Language Models

Korea University（高丽大学）；NEC Labs America（美国NEC实验室）；UC San Diego（加利福尼亚大学圣迭戈分校）；KAIST（韩国科学技术院）

Dohwan Ko, Sihyeon Kim, Yumin Suh, Vijay Kumar B. G, Minseo Yoon, Manmohan Chandraker, Hyunwoo J. Kim

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出ST-VLM，通过构建运动学指令微调数据集STKit和基准STKit-Bench，并利用4D重建生成伪标签，增强视觉语言模型的时空推理能力，在多个基准上超越基线，支持复杂多步推理。

摘要：

时空推理对于理解现实世界环境至关重要，广泛应用于自动驾驶和体育分析等领域。近期进展通过引入大规模数据提升了视觉语言模型（VLMs）的空间推理能力，但这些模型在处理移动物体的行驶距离和速度等运动学要素时仍存在困难。为弥补这一差距，我们构建了包含运动学指令微调的时空推理数据集和基准，分别称为STKit和STKit-Bench。它们包含带有3D标注的真实世界视频，详细描述了物体运动动态：行驶距离、速度、运动方向、物体间距离比较以及相对运动方向。为进一步将此类数据构建扩展到没有3D标签的视频，我们提出了一种自动流水线，利用真实尺度下的4D重建生成伪标签。借助用于时空推理的运动学指令微调数据，我们提出了ST-VLM，一种增强时空推理能力的视觉语言模型，在STKit-Bench上表现出色。此外，我们展示了ST-VLM在多种领域和任务上的稳健泛化能力，在其他时空基准（如ActivityNet、TVQA+）上优于基线模型。最后，通过将学到的时空推理与现有能力相结合，ST-VLM能够实现复杂的多步推理。项目页面：https://ikodoh.github.io/ST-VLM。

英文摘要：

Spatio-temporal reasoning is essential in understanding real-world environments in various fields, eg, autonomous driving and sports analytics. Recent advances have improved the spatial reasoning ability of Vision-Language Models (VLMs) by introducing large-scale data, but these models still struggle to analyze kinematic elements like traveled distance and speed of moving objects. To bridge this gap, we construct a spatio-temporal reasoning dataset and benchmark involving kinematic instruction tuning, referred to as STKit and STKit-Bench. They consist of real-world videos with 3D annotations, detailing object motion dynamics: traveled distance, speed, movement direction, inter-object distance comparisons, and relative movement direction. To further scale such data construction to videos without 3D labels, we propose an automatic pipeline to generate pseudo-labels using 4D reconstruction in real-world scale. With our kinematic instruction tuning data for spatio-temporal reasoning, we present ST-VLM, a VLM enhanced for spatio-temporal reasoning, which exhibits outstanding performance on STKit-Bench. Furthermore, we show that ST-VLM generalizes robustly across diverse domains and tasks, outperforming baselines on other spatio-temporal benchmarks (eg, ActivityNet, TVQA+). Finally, by integrating learned spatio-temporal reasoning with existing abilities, ST-VLM enables complex multi-step reasoning. Project page: https://ikodoh.github.io/ST-VLM.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。