arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.00295cs.CVcs.LG
收藏

Box2Flow:基于实例的视频动作流图

Box2Flow: Instance-based Action Flow Graphs from Videos

Rutgers University(罗格斯大学)

Jiatong Li, Kalliopi Basioti, Vladimir Pavlovic

AI总结:

Box2Flow提出一种基于单视频实例的步骤流图预测方法,通过提取边界框、预测步骤对边概率并用生成树算法构建流图,在MM-ReS和YouCookII上有效提取更准确的流图。

摘要:

网络上大量的程序性视频展示了如何完成各种任务。这些任务通常可以通过不同的方式和步骤顺序来完成,其中一些步骤可以同时执行,而另一些步骤则被限制必须按照特定顺序完成。流图可以用来表示任务的步骤关系。当前基于任务的方法试图为特定任务的所有可用视频学习一个单一的流图。提取出的流图往往过于抽象,无法捕捉详细的步骤描述。在本工作中,我们的目标是通过从单个视频中提取流图来学习准确且丰富的流图。我们提出了Box2Flow,这是一种基于实例的方法,用于从给定的程序性视频中预测步骤流图。具体来说,我们从视频中提取边界框,预测步骤对之间的成对边概率,并使用生成树算法构建流图。在MM-ReS和YouCookII上的实验表明,我们的方法能够有效地提取流图。

英文摘要:

A large amount of procedural videos on the web show how to complete various tasks. These tasks can often be accomplished in different ways and step orderings, with some steps able to be performed simultaneously, while others are constrained to be completed in a specific order. Flow graphs can be used to illustrate the step relationships of a task. Current task-based methods try to learn a single flow graph for all available videos of a specific task. The extracted flow graphs tend to be too abstract, failing to capture detailed step descriptions. In this work, our aim is to learn accurate and rich flow graphs by extracting them from a single video. We propose Box2Flow, an instance-based method to predict a step flow graph from a given procedural video. In detail, we extract bounding boxes from videos, predict pairwise edge probabilities between step pairs, and build the flow graph with a spanning tree algorithm. Experiments on MM-ReS and YouCookII show our method can extract flow graphs effectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑