arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2201.09169cs.CV
收藏

面向早期动作预测的丰富动作语义一致性知识

Rich Action-semantic Consistent Knowledge for Early Action Prediction

Beijing University of Posts and Telecommunications(北京邮电大学);Tsinghua University(清华大学)

Xiaoli Liu, Jianqin Yin, Di Guo, Huaping Liu

AI总结:

针对早期动作预测,提出RACK网络,通过划分视频并挖掘部分视频间的动作语义一致性知识,利用教师-学生框架和双向/单向语义图及蒸馏损失,在三个基准上达到最先进性能。

摘要:

早期动作预测(EAP)旨在从正在进行的视频中的部分动作执行中识别人类动作,这对于许多实际应用来说是一项重要任务。以往的大多数工作将部分或完整视频视为一个整体,忽略了视频中隐藏的丰富动作知识,即不同部分视频之间的语义一致性。相比之下,我们将原始的部分或完整视频进行划分,形成一系列新的部分视频,并挖掘这些在任意进度水平上演化的新部分视频之间的动作语义一致性知识(ASCK)。此外,我们提出了一种新颖的教师-学生框架下的丰富动作语义一致性知识网络(RACK),用于早期动作预测。首先,我们使用一个双流预训练模型来提取视频的特征。其次,我们将部分视频的RGB或光流特征视为节点,将它们的动作语义一致性视为边。接下来,我们为教师网络构建一个双向语义图,为学生网络构建一个单向语义图,以建模部分视频之间丰富的ASCK。均方误差(MSE)和最大均值差异(MMD)损失被纳入我们的蒸馏损失中,以丰富从教师网络到学生网络的部分视频的ASCK。最后,我们通过对不同子网络的logits求和并应用softmax层来获得最终预测。我们进行了广泛的实验和消融研究,证明了为EAP建模丰富ASCK的有效性。利用所提出的RACK,我们在三个基准上取得了最先进的性能。代码可在https://github.com/lily2lab/RACK.git获取。

英文摘要:

Early action prediction (EAP) aims to recognize human actions from a part of action execution in ongoing videos, which is an important task for many practical applications. Most prior works treat partial or full videos as a whole, ignoring rich action knowledge hidden in videos, i.e., semantic consistencies among different partial videos. In contrast, we partition original partial or full videos to form a new series of partial videos and mine the Action-Semantic Consistent Knowledge (ASCK) among these new partial videos evolving in arbitrary progress levels. Moreover, a novel Rich Action-semantic Consistent Knowledge network (RACK) under the teacher-student framework is proposed for EAP. Firstly, we use a two-stream pre-trained model to extract features of videos. Secondly, we treat the RGB or flow features of the partial videos as nodes and their action semantic consistencies as edges. Next, we build a bi-directional semantic graph for the teacher network and a single-directional semantic graph for the student network to model rich ASCK among partial videos. The MSE and MMD losses are incorporated as our distillation loss to enrich the ASCK of partial videos from the teacher to the student network. Finally, we obtain the final prediction by summering the logits of different subnetworks and applying a softmax layer. Extensive experiments and ablative studies have been conducted, demonstrating the effectiveness of modeling rich ASCK for EAP. With the proposed RACK, we have achieved state-of-the-art performance on three benchmarks. The code is available at https://github.com/lily2lab/RACK.git.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑