arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.17839cs.CVcs.AIcs.CLcs.LGcs.MM
收藏

从教学视频及其解说中学习过程感知的视频表示

Learning Procedure-aware Video Representation from Instructional Videos and Their Narrations

University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Meta AI

Yiwu Zhong, Licheng Yu, Yang Bai, Shangwen Li, Xueting Yan, Yin Li

AI总结:

本研究提出一种无需人工标注的方法,联合学习视频表示与深度概率模型以编码动作步骤概念及其时序依赖,显著提升了步骤分类与预测性能并支持零样本推理。

摘要:

互联网上丰富的教学视频及其解说为理解程序化活动提供了令人兴奋的途径。在本研究中,我们提出学习一种同时编码动作步骤及其时序顺序的视频表示,该方法基于大规模网络教学视频及其解说数据集,无需使用人工标注。我们的方法联合学习视频表示以编码单个步骤概念,并学习一个深度概率模型以捕获步骤顺序中的时序依赖关系和巨大的个体差异。我们通过实验证明,学习时序顺序不仅为程序推理提供了新能力,还增强了对单个步骤的识别。我们的模型在步骤分类(在COIN / EPIC-Kitchens上提升+2.8% / +3.3%)和步骤预测(在COIN上提升+7.4%)上显著推进了当前最优结果。此外,我们的模型在步骤分类和预测的零样本推理中取得了有前景的结果,并在为不完整程序预测多样且合理的步骤方面表现优异。代码可在https://github.com/facebookresearch/ProcedureVRL获取。

英文摘要:

The abundance of instructional videos and their narrations over the Internet offers an exciting avenue for understanding procedural activities. In this work, we propose to learn video representation that encodes both action steps and their temporal ordering, based on a large-scale dataset of web instructional videos and their narrations, without using human annotations. Our method jointly learns a video representation to encode individual step concepts, and a deep probabilistic model to capture both temporal dependencies and immense individual variations in the step ordering. We empirically demonstrate that learning temporal ordering not only enables new capabilities for procedure reasoning, but also reinforces the recognition of individual steps. Our model significantly advances the state-of-the-art results on step classification (+2.8% / +3.3% on COIN / EPIC-Kitchens) and step forecasting (+7.4% on COIN). Moreover, our model attains promising results in zero-shot inference for step classification and forecasting, as well as in predicting diverse and plausible steps for incomplete procedures. Our code is available at https://github.com/facebookresearch/ProcedureVRL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑