arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.05166cs.CV
收藏

TAEC:基于时间感知嵌入和聚类的无监督动作分割

TAEC: Unsupervised Action Segmentation with Temporal-Aware Embedding and Clustering

Graz University of Technology(格拉茨技术大学);Christian Doppler Laboratory for Semantic 3D Computer Vision(克里斯蒂安·多普勒语义三维计算机视觉实验室);Max-Planck-Institute for Informatics(马克斯·普朗克信息学研究所);Goethe University Frankfurt(法兰克福歌德大学)

Wei Lin, Anna Kukleva, Horst Possegger, Hilde Kuehne, Horst Bischof

摘要:

最近,未修剪视频中的时间动作分割得到了越来越多的关注。然而,注释动作类和框架边界非常耗时且成本密集,尤其是在大规模数据集上。为了解决这个问题,我们提出了一种无监督的方法来学习动作类从未经修剪的视频序列。特别是,我们提出了一个时间嵌入网络,它结合了相对时间预测,特征重建和序列到序列学习,以保持视频特征的空间布局和顺序性质。对这些嵌入式特征表示的两步聚类管道允许我们在视频内以及跨视频执行时间一致性。基于所识别的聚类,我们将视频解码成对应于语义上有意义的动作类的连贯的时间段。我们对三个具有挑战性的数据集的评估显示了每个组件的影响,并进一步展示了我们最先进的无监督动作分割结果。

英文摘要:

Temporal action segmentation in untrimmed videos has gained increased attention recently. However, annotating action classes and frame-wise boundaries is extremely time consuming and cost intensive, especially on large-scale datasets. To address this issue, we propose an unsupervised approach for learning action classes from untrimmed video sequences. In particular, we propose a temporal embedding network that combines relative time prediction, feature reconstruction, and sequence-to-sequence learning, to preserve the spatial layout and sequential nature of the video features. A two-step clustering pipeline on these embedded feature representations then allows us to enforce temporal consistency within, as well as across videos. Based on the identified clusters, we decode the video into coherent temporal segments that correspond to semantically meaningful action classes. Our evaluation on three challenging datasets shows the impact of each component and, furthermore, demonstrates our state-of-the-art unsupervised action segmentation results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑