通过叙述学习在视频中定位教学文章
Learning to Ground Instructional Articles in Videos through Narrations
AI总结：
本文提出无监督融合帧、叙述与 wikiHow 步骤描述的三模态对齐方法，实现教学步骤全局时间定位，并构建 HT-Step 验证其显著优势。
摘要：
本文提出了一种在带叙述的操作演示视频中定位程序性活动步骤的方法。为解决大规模标注数据稀缺的问题，我们从包含大量程序性任务教学文章的语言知识库 wikiHow 中获取步骤描述。无需任何形式的人工监督，我们的模型通过匹配帧、叙述和步骤描述三种模态，学习将教学文章中的步骤在时间上定位于操作演示视频。具体而言，该方法通过融合两条不同路径的信息将步骤与视频对齐：i）步骤描述与帧的直接对齐；ii）通过组合步骤到叙述以及叙述到视频的对应关系而获得的间接对齐。值得注意的是，我们的方法利用顺序信息，一次性对一篇文章中的所有步骤执行全局时间定位，并使用经迭代细化和严格过滤的步骤伪标签进行训练。为验证模型，我们引入了一个新的评估基准 HT-Step：它通过对 HowTo100M 的一个124小时子集进行人工标注构建，步骤来源于 wikiHow 文章，测试服务器可在 https://eval.ai/web/challenges/challenge-page/2082 访问。在该基准上的实验以及在 CrossTask 上的零样本评估表明，与若干基线和先前工作相比，我们的多模态对齐带来了显著提升。最后，我们证明用于叙述到视频匹配的内部模块在 HTM-Align 叙述-视频对齐基准上大幅优于当前最先进方法。
英文摘要：
In this paper we present an approach for localizing steps of procedural activities in narrated how-to videos. To deal with the scarcity of labeled data at scale, we source the step descriptions from a language knowledge base (wikiHow) containing instructional articles for a large variety of procedural tasks. Without any form of manual supervision, our model learns to temporally ground the steps of procedural articles in how-to videos by matching three modalities: frames, narrations, and step descriptions. Specifically, our method aligns steps to video by fusing information from two distinct pathways: i) {\em direct} alignment of step descriptions to frames, ii) {\em indirect} alignment obtained by composing steps-to-narrations with narrations-to-video correspondences. Notably, our approach performs global temporal grounding of all steps in an article at once by exploiting order information, and is trained with step pseudo-labels which are iteratively refined and aggressively filtered. In order to validate our model we introduce a new evaluation benchmark -- HT-Step -- obtained by manually annotating a 124-hour subset of HowTo100M\footnote{A test server is accessible at \url{https://eval.ai/web/challenges/challenge-page/2082}.} with steps sourced from wikiHow articles. Experiments on this benchmark as well as zero-shot evaluations on CrossTask demonstrate that our multi-modality alignment yields dramatic gains over several baselines and prior works. Finally, we show that our inner module for matching narration-to-video outperforms by a large margin the state of the art on the HTM-Align narration-video alignment benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
