什么、何时、何地？——从叙述指令中学习未修剪多动作视频的自监督时空定位
What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions
AI总结：
提出一种仅用视频和字幕监督训练的自监督时空定位框架，结合局部与全局表示学习，并在新基准上超越现有基线。
摘要：
时空定位描述的是仅基于口头描述在空间和时间上定位事件的任务，例如在视频数据中。该任务的模型通常使用人工标注的句子和边界框监督进行训练。本研究从多模态监督的角度解决该任务，提出了一种仅依靠松散的视频和字幕监督进行时空动作定位的框架，无需人工标注。为此，我们将侧重于利用细粒度空间信息的局部表示学习与捕获高层表示的全局表示编码相结合，并将两者纳入一种联合方法中。为了在真实场景中评估这一具有挑战性的任务，我们提出了一个新的基准数据集，为超过5000个事件的长、未修剪、多动作教学视频提供了密集的时空定位标注。我们在所提出的和标准的下游任务上评估了所提出的方法和其他方法，结果表明，我们的方法在多种设置下（包括空间、时间以及未修剪多动作时空定位）均优于现有基线。
英文摘要：
Spatio-temporal grounding describes the task of localizing events in space and time, e.g., in video data, based on verbal descriptions only. Models for this task are usually trained with human-annotated sentences and bounding box supervision. This work addresses this task from a multimodal supervision perspective, proposing a framework for spatio-temporal action grounding trained on loose video and subtitle supervision only, without human annotation. To this end, we combine local representation learning, which focuses on leveraging fine-grained spatial information, with a global representation encoding that captures higher-level representations and incorporates both in a joint approach. To evaluate this challenging task in a real-life setting, a new benchmark dataset is proposed providing dense spatio-temporal grounding annotations in long, untrimmed, multi-action instructional videos for over 5K events. We evaluate the proposed approach and other methods on the proposed and standard downstream tasks showing that our method improves over current baselines in various settings, including spatial, temporal, and untrimmed multi-action spatio-temporal grounding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
