为什么不使用你的教科书？教学视频的知识增强过程规划
Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videos
AI总结：
针对教学视频过程规划中的因果约束与多方案变化问题，本文提出 KEPP，利用从训练数据构建的概率性过程知识图谱增强智能体，并在三个数据集上以最少监督达到最优性能。
摘要：
在本文中，我们探索了智能体构建逻辑化行动步骤序列，从而形成策略性过程计划的能力。正如现实生活中的教学视频所描绘的那样，该计划对于从初始视觉观察导航到目标视觉结果至关重要。现有工作通过广泛利用数据集中可用的各种信息来源（例如大量中间视觉观察、过程名称或自然语言逐步指令）作为特征或监督信号，已经取得了部分成功。然而，由于步骤排序中隐含的因果约束以及多种可行计划固有的可变性，该任务仍然十分艰巨。为了解决以往工作忽视的这些复杂问题，我们提议通过向智能体注入过程性知识来增强其能力。该知识来源于训练过程计划，并被结构化为有向加权图，使智能体能够更好地应对步骤排序及其潜在变化的复杂性。我们将该方法命名为 KEPP，这是一种新颖的知识增强过程规划系统，它利用从训练数据中提取的概率性过程知识图谱，实际上充当训练领域的综合性教科书。在三个广泛使用的数据集上、不同复杂度的设置下进行的实验评估表明，KEPP 仅需最少的监督即可取得优越的、最先进的结果。
英文摘要：
In this paper, we explore the capability of an agent to construct a logical sequence of action steps, thereby assembling a strategic procedural plan. This plan is crucial for navigating from an initial visual observation to a target visual outcome, as depicted in real-life instructional videos. Existing works have attained partial success by extensively leveraging various sources of information available in the datasets, such as heavy intermediate visual observations, procedural names, or natural language step-by-step instructions, for features or supervision signals. However, the task remains formidable due to the implicit causal constraints in the sequencing of steps and the variability inherent in multiple feasible plans. To tackle these intricacies that previous efforts have overlooked, we propose to enhance the capabilities of the agent by infusing it with procedural knowledge. This knowledge, sourced from training procedure plans and structured as a directed weighted graph, equips the agent to better navigate the complexities of step sequencing and its potential variations. We coin our approach KEPP, a novel Knowledge-Enhanced Procedure Planning system, which harnesses a probabilistic procedural knowledge graph extracted from training data, effectively acting as a comprehensive textbook for the training domain. Experimental evaluations across three widely-used datasets under settings of varying complexity reveal that KEPP attains superior, state-of-the-art results while requiring only minimal supervision.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
