面向长形式时空视频定位
Towards Long-Form Spatio-Temporal Video Grounding
AI总结：
本文提出ART-STVG方法，针对长时视频中的目标定位问题，通过自回归Transformer架构和记忆银行设计，提升对长视频中时空信息的处理能力，实验表明其在长形式STVG任务中优于现有方法。
摘要：
在现实场景中，视频可能持续数分钟甚至数小时。然而，现有时空视频定位（STVG）研究主要针对短视频（几十秒），通常不足一分钟，限制了实际应用。本文探索长形式STVG（LF-STVG），旨在定位长时视频中的目标。与短视频相比，长视频包含更长的时间跨度和更多无关信息，使现有处理所有帧的方法难以应对。为此，我们提出ART-STVG架构，将视频视为流式输入，按顺序处理帧，提高长视频处理效率。为建模时空上下文，我们设计了空间和时间记忆银行，并应用于解码器。由于不同时刻的记忆不一定相关，我们引入简单有效的记忆选择策略，为解码器提供更相关的信息，显著提升性能。此外，我们提出级联时空设计，将空间解码器连接到时间解码器，使精细的空间线索辅助长视频中的复杂时间定位。在扩展的LF-STVG数据集上进行的实验表明，ART-STVG在长形式STVG任务中显著优于现有方法，同时在传统短形式STVG任务中也取得竞争性性能。我们的代码见：https://github.com/HengLan/ART-STVG。
英文摘要：
In real scenarios, videos can span several minutes or even hours. However, existing research on spatio-temporal video grounding (STVG), given a textual query, mainly focuses on localizing targets in short videos of tens of seconds, typically less than one minute, which limits real-world applications. In this paper, we explore Long-Form STVG (LF-STVG), which aims to locate targets in long-term videos. Compared with short videos, long-term videos contain much longer temporal spans and more irrelevant information, making it difficult for existing STVG methods that process all frames at once. To address this challenge, we propose an AutoRegressive Transformer architecture for LF-STVG, termed ART-STVG. Unlike conventional STVG methods that require the entire video sequence to make predictions at once, ART-STVG treats the video as streaming input and processes frames sequentially, enabling efficient handling of long videos. To model spatio-temporal context, we design spatial and temporal memory banks and apply them to the decoders. Since memories from different moments are not always relevant to the current frame, we introduce simple yet effective memory selection strategies to provide more relevant information to the decoders, significantly improving performance. Furthermore, instead of parallel spatial and temporal localization, we propose a cascaded spatio-temporal design that connects the spatial decoder to the temporal decoder, allowing fine-grained spatial cues to assist complex temporal localization in long videos. Experiments on newly extended LF-STVG datasets show that ART-STVG significantly outperforms state-of-the-art methods, while achieving competitive performance on conventional short-form STVG. Our code is at: https://github.com/HengLan/ART-STVG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
