StepFormer：教学视频中的自监督步骤发现与定位
StepFormer: Self-supervised Step Discovery and Localization in Instructional Videos
AI总结：
提出自监督 StepFormer，仅利用自动字幕和顺序感知损失发现并定位教学视频中的关键步骤，在三个基准上显著优于无监督与弱监督方法，并具备零样本多步骤定位能力。
摘要：
教学视频是从人类演示中学习程序性任务的重要资源。然而，此类视频中的教学步骤通常短暂且稀疏，视频的大部分内容与该流程无关。这促使人们需要在此类视频中对教学步骤进行时间定位，即所谓的关键步骤定位任务。传统的关键步骤定位方法需要视频级人工标注，因此无法扩展到大型数据集。在本工作中，我们在没有人工监督的情况下解决该问题，并提出 StepFormer，这是一种用于发现并定位视频中教学步骤的自监督模型。StepFormer 是一个 transformer 解码器，通过可学习查询关注视频，并生成一系列槽位以捕获视频中的关键步骤。我们在大型教学视频数据集上训练系统，将自动生成的字幕作为唯一监督来源。具体而言，我们使用文本旁白序列监督系统，并采用一种能够过滤无关短语的顺序感知损失函数。我们表明，在三个具有挑战性的基准上，本模型在步骤检测和定位方面大幅优于此前所有无监督和弱监督方法。此外，本模型展现出一种涌现属性，能够解决零样本多步骤定位，并在该任务上优于所有相关基线。
英文摘要：
Instructional videos are an important resource to learn procedural tasks from human demonstrations. However, the instruction steps in such videos are typically short and sparse, with most of the video being irrelevant to the procedure. This motivates the need to temporally localize the instruction steps in such videos, i.e. the task called key-step localization. Traditional methods for key-step localization require video-level human annotations and thus do not scale to large datasets. In this work, we tackle the problem with no human supervision and introduce StepFormer, a self-supervised model that discovers and localizes instruction steps in a video. StepFormer is a transformer decoder that attends to the video with learnable queries, and produces a sequence of slots capturing the key-steps in the video. We train our system on a large dataset of instructional videos, using their automatically-generated subtitles as the only source of supervision. In particular, we supervise our system with a sequence of text narrations using an order-aware loss function that filters out irrelevant phrases. We show that our model outperforms all previous unsupervised and weakly-supervised approaches on step detection and localization by a large margin on three challenging benchmarks. Moreover, our model demonstrates an emergent property to solve zero-shot multi-step localization and outperforms all relevant baselines at this task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
