面向教学视频理解的过程感知预训练
Procedure-Aware Pretraining for Instructional Video Understanding
AI总结：
本文提出基于过程知识图（PKG）的预训练模型 Paprika，利用未标注视频和文本知识库生成伪标签进行预训练，在 COIN 和 CrossTask 的多项过程理解任务中准确率最高提升 11.23%。
摘要：
我们的目标是学习一种对下游教学视频中的过程理解任务有用的视频表示。由于可用的标注数据较少，过程理解的一个关键挑战是能够从未标注视频中提取过程知识，例如任务的身份（如“制作拿铁”）、其步骤（如“倒牛奶”），或在任务执行部分进展后潜在的下一步骤。我们的主要见解是，教学视频描绘了在相同或不同任务实例之间重复的步骤序列，这种结构可以由过程知识图（PKG）很好地表示，其中节点是离散步骤，边连接在教学活动中顺序出现的步骤。然后，该图可用于生成伪标签，以训练一种以更易获取的形式编码过程知识、从而泛化到多个过程理解任务的视频表示。我们通过结合基于文本的过程知识数据库和未标注教学视频语料库来构建 PKG，随后利用它生成训练伪标签，并提出了四个新颖的预训练目标。我们将这种基于 PKG 的预训练过程及由此产生的模型称为 Paprika，即面向指令知识获取的过程感知预训练。我们在 COIN 和 CrossTask 上对 Paprika 进行了过程理解任务评估，例如任务识别、步骤识别和步骤预测。Paprika 产生的视频表示优于现有技术：在 12 项评估设置中准确率提升高达 11.23%。实现代码可在 https://github.com/salesforce/paprika 获取。
英文摘要：
Our goal is to learn a video representation that is useful for downstream procedure understanding tasks in instructional videos. Due to the small amount of available annotations, a key challenge in procedure understanding is to be able to extract from unlabeled videos the procedural knowledge such as the identity of the task (e.g., 'make latte'), its steps (e.g., 'pour milk'), or the potential next steps given partial progress in its execution. Our main insight is that instructional videos depict sequences of steps that repeat between instances of the same or different tasks, and that this structure can be well represented by a Procedural Knowledge Graph (PKG), where nodes are discrete steps and edges connect steps that occur sequentially in the instructional activities. This graph can then be used to generate pseudo labels to train a video representation that encodes the procedural knowledge in a more accessible form to generalize to multiple procedure understanding tasks. We build a PKG by combining information from a text-based procedural knowledge database and an unlabeled instructional video corpus and then use it to generate training pseudo labels with four novel pre-training objectives. We call this PKG-based pre-training procedure and the resulting model Paprika, Procedure-Aware PRe-training for Instructional Knowledge Acquisition. We evaluate Paprika on COIN and CrossTask for procedure understanding tasks such as task recognition, step recognition, and step forecasting. Paprika yields a video representation that improves over the state of the art: up to 11.23% gains in accuracy in 12 evaluation settings. Implementation is available at https://github.com/salesforce/paprika.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
