面向弱监督时空视频定位的上下文自步学习
Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding
AI总结：
针对弱监督时空视频定位中检测模型时间预测不一致和复杂查询理解不足等问题，提出融合TPG、CRG与SPS的CoSPaL框架，以渐进方式提升复杂场景定位能力。
摘要：
在本工作中，我们聚焦于弱监督时空视频定位（WSTVG）。这是一项多模态任务，旨在没有边界框监督的情况下，根据文本查询在时空上定位特定主体。受近期用于定位任务的多模态基础模型进展的启发，我们首先探索了最先进的目标检测模型用于WSTVG的潜力。尽管这些模型具有强大的零样本能力，但我们的适配揭示了显著局限，包括时间预测不一致、对复杂查询理解不足，以及适应困难场景时面临挑战。我们提出CoSPaL（Contextual Self-Paced Learning，上下文自步学习），这是一种旨在克服上述局限的新颖方法。CoSPaL集成了三个核心组件：（1）管短语定位（TPG），通过将文本查询与管（tubelets）关联来引入时空预测；（2）上下文指涉定位（CRG），通过提取上下文信息来改进随时间变化的目标识别，从而增强对复杂查询的理解；（3）自步场景理解（SPS），这是一种逐步增加任务难度的训练范式，使模型能够通过从粗粒度理解过渡到细粒度理解来适应复杂场景。
英文摘要：
In this work, we focus on Weakly Supervised Spatio-Temporal Video Grounding (WSTVG). It is a multimodal task aimed at localizing specific subjects spatio-temporally based on textual queries without bounding box supervision. Motivated by recent advancements in multi-modal foundation models for grounding tasks, we first explore the potential of state-of-the-art object detection models for WSTVG. Despite their robust zero-shot capabilities, our adaptation reveals significant limitations, including inconsistent temporal predictions, inadequate understanding of complex queries, and challenges in adapting to difficult scenarios. We propose CoSPaL (Contextual Self-Paced Learning), a novel approach which is designed to overcome these limitations. CoSPaL integrates three core components: (1) Tubelet Phrase Grounding (TPG), which introduces spatio-temporal prediction by linking textual queries to tubelets; (2) Contextual Referral Grounding (CRG), which improves comprehension of complex queries by extracting contextual information to refine object identification over time; and (3) Self-Paced Scene Understanding (SPS), a training paradigm that progressively increases task difficulty, enabling the model to adapt to complex scenarios by transitioning from coarse to fine-grained understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
