Pocket-STVG：用于时空视频定位的轻量级架构
Pocket-STVG: lightweight architecture for Spatio-Temporal Video Grounding
浏览论文内容
中文总结 AI 辅助
提出轻量级级联架构Pocket-STVG，结合高效预训练组件实现时空视频定位，在弱监督和零样本设置下以少于90M参数达到与弱监督方法相当的性能，并优于早期零样本方法，实现性能与效率的有利权衡。
中文摘要 AI 辅助
时空视频定位（STVG）旨在定位视频中与自然语言查询相对应的时空管。尽管近期方法在全监督、弱监督和零样本设置下取得了强劲性能，但它们通常依赖于计算成本高昂的架构、复杂的训练流程或多模态大语言模型。我们提出了Pocket-STVG（P-STVG），一种轻量级级联架构，通过结合高效的预训练组件而非大型端到端模型来解决STVG问题。P-STVG集成了基于MobileViCLIP的时序感知视频编码器、源自MDETR的空间编码-解码器以及一个共享的对齐文本编码器。时序定位通过轻量级1D U-Net或简单的阈值策略实现，使同一框架能够在弱监督和零样本设置下运行。此外，视频表示独立于查询进行预计算，从而产生一个索引友好的流程，用于高效推理和大规模视频集合。尽管参数少于90M，P-STVG的性能与弱监督方法相当，并优于早期的零样本方法，而其内存和计算成本仅为它们的一小部分，为STVG建立了有利的性能-效率权衡。
英文摘要
Spatio-Temporal Video Grounding (STVG) aims to localize the spatio-temporal tube in a video corresponding to a natural language query. While recent methods achieve strong performance in fully supervised, weakly supervised, and zero-shot settings, they typically rely on computationally expensive architectures, complex training pipelines, or multimodal large language models. We present Pocket-STVG (P-STVG), a lightweight cascade architecture that addresses STVG by combining efficient pre-trained components instead of large end-to-end models. P-STVG integrates a temporal-aware video encoder based on MobileViCLIP, a spatial encoder-decoder derived from MDETR, and a shared aligned text encoder. Temporal localization is performed through either a lightweight 1D U-Net or a simple thresholding strategy, enabling the same framework to operate in both weakly supervised and zero-shot settings. Furthermore, video representations are precomputed independently of the query, yielding an indexing-friendly pipeline for efficient inference and large-scale video collections. Despite requiring fewer than 90M parameters, P-STVG performs on par with weakly supervised methods and improves on earlier zero-shot approaches at a fraction of their memory and computational cost, establishing a favorable performance-efficiency trade-off for STVG.
发表机构
- Samsung AI Center（三星人工智能中心）
- IFTR, Polish Academy of Sciences（波兰科学院基础技术研究所）
机构由 AI 辅助整理，请以论文原文为准。
