arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2309.05972cs.CVcs.RO
收藏

通过逐帧离散特征自监督提取人体运动结构

Self-supervised Extraction of Human Motion Structures via Frame-wise Discrete Features

Tokyo University of Agriculture and Technology(东京农工大学);National Institute of Advanced Industrial Science and Technology(产业技术综合研究所);Osaka University(大阪大学)

Tetsuya Abe, Ryusuke Sagawa, Ko Ayusawa, Wataru Takano

AI总结:

提出一种自监督编码器-解码器模型,利用自注意力层和向量聚类块提取逐帧离散运动码本,并通过图可视化码间关系,在多识别任务中达到与任务优化方法相当的性能。

摘要:

本文提出了一种编码器-解码器模型,用于以自监督方式提取由逐帧离散特征表示的人体运动结构。在所提出的方法中,特征被提取为运动码本中的码,且不使用人类知识,这些码之间的关系可以在图上可视化。由于与捕获的帧率相比,这些码在时间上预期是稀疏的,并且可以被多个序列共享,因此所提出的网络模型也解决了对训练约束的需求。具体而言,该模型由自注意力层和一个向量聚类块组成。注意力层有助于找到稀疏关键帧和作为运动码的离散特征,这些特征随后通过向量聚类被提取。约束被实现为训练损失,以便相同的运动码可以尽可能连续,并且可以被多个序列共享。此外,我们提出使用因果自注意力作为一种计算方法,用于对由大量帧组成的长序列进行注意力计算。在我们的实验中,运动码的稀疏结构被用来编译一个图,该图有助于可视化码之间的关系以及序列之间的差异。然后,我们通过将提取的运动码应用于多个识别任务来评估其有效性,并发现在线性探测下可以实现与任务优化方法相当的性能水平。

英文摘要:

The present paper proposes an encoder-decoder model for extracting the structures of human motions represented by frame-wise discrete features in a self-supervised manner. In the proposed method, features are extracted as codes in a motion codebook without the use of human knowledge, and the relationship between these codes can be visualized on a graph. Since the codes are expected to be temporally sparse compared to the captured frame rate and can be shared by multiple sequences, the proposed network model also addresses the need for training constraints. Specifically, the model consists of self-attention layers and a vector clustering block. The attention layers contribute to finding sparse keyframes and discrete features as motion codes, which are then extracted by vector clustering. The constraints are realized as training losses so that the same motion codes can be as contiguous as possible and can be shared by multiple sequences. In addition, we propose the use of causal self-attention as a method by which to calculate attention for long sequences consisting of numerous frames. In our experiments, the sparse structures of motion codes were used to compile a graph that facilitates visualization of the relationship between the codes and the differences between sequences. We then evaluated the effectiveness of the extracted motion codes by applying them to multiple recognition tasks and found that performance levels comparable to task-optimized methods could be achieved by linear probing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑