教学视频分割中的时序分析
A Closer Look at Temporal Ordering in the Segmentation of Instructional Videos
摘要:
理解执行任务所需的步骤是AI系统的一项重要技能。从教学视频中学习这些步骤涉及两个子问题:(i)识别顺序出现的片段的时间边界和(ii)用自然语言总结这些步骤。我们将此任务称为过程分割和总结(PSS)。在本文中,我们仔细研究PSS,并提出了三个根本的改进,目前的方法。分割任务至关重要,因为生成正确的摘要需要正确识别程序的每个步骤。然而,目前的分割指标往往高估了分割质量,因为他们不考虑段的时间顺序。在我们的第一个贡献中,我们提出了一个新的分割度量,考虑到段的顺序,给出了一个更可靠的衡量一个给定的预测分割的准确性。目前的PSS方法通常通过提出分段、将它们与地面实况匹配并计算损失来训练。然而,很像分割度量,现有的匹配算法不考虑候选片段和地面实况之间的映射的时间顺序。在我们的第二个贡献,我们提出了一个匹配算法,约束的时间顺序段映射,也是可微的。最后,我们为PSS引入了多模态特征训练,这进一步改进了分割。我们评估我们的方法在两个教学视频数据集(YouCook2和Tasty),并观察到改进的最先进的$\sim7\%$和$\sim2.5\%$的程序分割和总结,分别。
英文摘要:
Understanding the steps required to perform a task is an important skill for AI systems. Learning these steps from instructional videos involves two subproblems: (i) identifying the temporal boundary of sequentially occurring segments and (ii) summarizing these steps in natural language. We refer to this task as Procedure Segmentation and Summarization (PSS). In this paper, we take a closer look at PSS and propose three fundamental improvements over current methods. The segmentation task is critical, as generating a correct summary requires each step of the procedure to be correctly identified. However, current segmentation metrics often overestimate the segmentation quality because they do not consider the temporal order of segments. In our first contribution, we propose a new segmentation metric that takes into account the order of segments, giving a more reliable measure of the accuracy of a given predicted segmentation. Current PSS methods are typically trained by proposing segments, matching them with the ground truth and computing a loss. However, much like segmentation metrics, existing matching algorithms do not consider the temporal order of the mapping between candidate segments and the ground truth. In our second contribution, we propose a matching algorithm that constrains the temporal order of segment mapping, and is also differentiable. Lastly, we introduce multi-modal feature training for PSS, which further improves segmentation. We evaluate our approach on two instructional video datasets (YouCook2 and Tasty) and observe an improvement over the state-of-the-art of $\sim7\%$ and $\sim2.5\%$ for procedure segmentation and summarization, respectively.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。