可微任务图学习：来自第一人称视频的程序性活动表示与在线错误检测
Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos
AI总结：
本文提出可微任务图学习框架，通过边权重最大似然优化实现梯度学习，提升任务图预测与第一人称视频在线错误检测性能。
摘要：
程序性活动是旨在实现特定目标的关键步骤序列。它们对于构建能够有效协助用户的智能体至关重要。在此背景下，任务图已成为程序性活动的人类可理解表示，编码了关键步骤之间的偏序关系。以往工作通常依赖手工制定的流程来从视频中提取任务图，而本文提出一种基于边权重直接最大似然优化的方法，该方法支持基于梯度的任务图学习，并能自然地接入神经网络架构。在 CaptainCook4D 数据集上的实验表明，我们的方法能够通过观察动作序列预测准确的任务图，相比以往方法提升了 +16.7%。得益于所提出框架的可微性，我们还引入了一种基于特征的方法，旨在根据关键步骤的文本或视频嵌入预测任务图，并在其中观察到涌现的视频理解能力。研究还表明，用我们的方法学习到的任务图能够显著增强程序性第一人称视频中的在线错误检测，在 Assembly101-O 和 EPIC-Tent-O 数据集上分别取得 +19.8% 和 +7.5% 的显著增益。复现实验的代码可在 https://github.com/fpv-iplab/Differentiable-Task-Graph-Learning 获取。
英文摘要：
Procedural activities are sequences of key-steps aimed at achieving specific goals. They are crucial to build intelligent agents able to assist users effectively. In this context, task graphs have emerged as a human-understandable representation of procedural activities, encoding a partial ordering over the key-steps. While previous works generally relied on hand-crafted procedures to extract task graphs from videos, in this paper, we propose an approach based on direct maximum likelihood optimization of edges' weights, which allows gradient-based learning of task graphs and can be naturally plugged into neural network architectures. Experiments on the CaptainCook4D dataset demonstrate the ability of our approach to predict accurate task graphs from the observation of action sequences, with an improvement of +16.7% over previous approaches. Owing to the differentiability of the proposed framework, we also introduce a feature-based approach, aiming to predict task graphs from key-step textual or video embeddings, for which we observe emerging video understanding abilities. Task graphs learned with our approach are also shown to significantly enhance online mistake detection in procedural egocentric videos, achieving notable gains of +19.8% and +7.5% on the Assembly101-O and EPIC-Tent-O datasets. Code for replicating experiments is available at https://github.com/fpv-iplab/Differentiable-Task-Graph-Learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
