arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2304.06841cs.CVcs.LG
收藏

利用局部与全局特征无监督学习进行视频对齐

Video alignment using unsupervised learning of local and global features

Sharif University of Technology(谢里夫理工大学)

Niloufar Fakhfour, Mohammad ShahverdiKondori, Sajjad Hashembeiki, Mohammadjavad Norouzi, Hoda Mohammadzade

AI总结:

提出融合行人检测、姿态估计与 VGG 特征的无监督视频对齐方法,并用 DDTW 对齐多维时间序列,在 Penn action 与 UCF101 子集上优于现有方法。

摘要:

本文研究视频对齐问题,即对包含相似动作的一对视频进行帧匹配的过程。视频对齐的主要挑战在于,尽管两个视频在执行过程和外观上存在差异,仍需建立准确的对应关系。我们提出一种利用帧的全局和局部特征的无监督对齐方法。具体而言,我们借助三种机器视觉工具为每个视频帧引入有效特征:行人检测、姿态估计和 VGG 网络。随后对这些特征进行处理和组合,构建表示视频的多维时间序列。所得时间序列通过一种名为对角化动态时间规整(Diagonalized Dynamic Time Warping,DDTW)的新型动态时间规整方法,用于对齐相同动作的视频。我们方法的主要优势是无需训练,因此可适用于任何新型动作,而不必为其收集训练样本。此外,我们的方法还可用于仅含少量标注视频的数据集中动作阶段的逐帧标注。在评估方面,我们在 Penn action 数据集和 UCF101 数据集子集上考虑了视频同步和阶段分类任务。同时,为有效评估视频同步任务,我们提出一种称为包围面积误差(Enclosed Area Error,EAE)的新指标。结果表明,我们的方法优于 TCC 等以往的最先进方法,以及其他自监督和弱监督方法。

英文摘要:

In this paper, we tackle the problem of video alignment, the process of matching the frames of a pair of videos containing similar actions. The main challenge in video alignment is that accurate correspondence should be established despite the differences in the execution processes and appearances between the two videos. We introduce an unsupervised method for alignment that uses global and local features of the frames. In particular, we introduce effective features for each video frame by means of three machine vision tools: person detection, pose estimation, and VGG network. Then the features are processed and combined to construct a multidimensional time series that represent the video. The resulting time series are used to align videos of the same actions using a novel version of dynamic time warping named Diagonalized Dynamic Time Warping(DDTW). The main advantage of our approach is that no training is required, which makes it applicable for any new type of action without any need to collect training samples for it. Additionally, our approach can be used for framewise labeling of action phases in a dataset with only a few labeled videos. For evaluation, we considered video synchronization and phase classification tasks on the Penn action and subset of UCF101 datasets. Also, for an effective evaluation of the video synchronization task, we present a new metric called Enclosed Area Error(EAE). The results show that our method outperforms previous state-of-the-art methods, such as TCC, and other self-supervised and weakly supervised methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑