arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1912.06992cs.CV
收藏

Action Genome:作为时空场景图合成的动作

Action Genome: Actions as Composition of Spatio-temporal Scene Graphs

Stanford University(斯坦福大学)

Jingwei Ji, Ranjay Krishna, Li Fei-Fei, Juan Carlos Niebles

摘要:

动作识别通常将动作和活动视为视频中发生的单一事件。然而,来自认知科学和神经科学的证据表明,人们积极地将活动编码为一致的分层部分结构。然而,在计算机视觉中,很少有探索的表示编码的事件partonomies已经取得。受事件的原型单位是一个动作-对象交互的证据的启发,我们引入动作基因组,一种将动作分解成时空场景图的表示。动作基因组捕获动作发生时对象之间的变化及其成对关系。它包含10 K视频,其中包含0.4M对象和1.7M视觉关系注释。与动作基因组,我们扩展了现有的动作识别模型,将场景图作为时空特征银行,以实现更好的性能上的字谜数据集。接下来,通过分解和学习的视觉关系,导致在一个动作的时间变化,我们证明了一个分层的事件分解的效用,使Few-Shot动作识别,实现42.7%的mAP使用少至10个例子。最后,我们基准现有的场景图模型的时空场景图预测的新任务。

英文摘要:

Action recognition has typically treated actions and activities as monolithic events that occur in videos. However, there is evidence from Cognitive Science and Neuroscience that people actively encode activities into consistent hierarchical part structures. However in Computer Vision, few explorations on representations encoding event partonomies have been made. Inspired by evidence that the prototypical unit of an event is an action-object interaction, we introduce Action Genome, a representation that decomposes actions into spatio-temporal scene graphs. Action Genome captures changes between objects and their pairwise relationships while an action occurs. It contains 10K videos with 0.4M objects and 1.7M visual relationships annotated. With Action Genome, we extend an existing action recognition model by incorporating scene graphs as spatio-temporal feature banks to achieve better performance on the Charades dataset. Next, by decomposing and learning the temporal changes in visual relationships that result in an action, we demonstrate the utility of a hierarchical event decomposition by enabling few-shot action recognition, achieving 42.7% mAP using as few as 10 examples. Finally, we benchmark existing scene graph models on the new task of spatio-temporal scene graph prediction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑