arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.28173cs.CV
收藏

动作基元:人体运动的自监督层次化表示

Action Motifs: Self-Supervised Hierarchical Representation of Human Body Movements

Kyoto University(京都大学);Kyoto Institute of Technology(京都理工学院);RIKEN(理化学研究所)

Genki Kinoshita, Shu Nakamura, Ryo Kawahara, Shohei Nobuhara, Yasutomo Kawanishi, Ko Nishino

AI总结:

提出一种层次化表示方法,通过自监督学习从人体姿态数据中提取动作原子和动作基元,用于动作识别、运动预测和运动插值等任务。

摘要:

有效的人类行为建模需要一种能够利用其组合性的人体运动表示。我们提出了一种层次化表示,包括捕获原子关节运动的动作原子和由它们的时间组合形成的动作基元,这些基元编码了在不同整体人类动作中发现的相似身体运动。我们推导出A4Mer,一种嵌套的潜在Transformer,以完全自监督的方式从人体姿态数据中学习这种层次化表示。A4Mer将3D姿态序列分割成可变长度的片段,并将每个片段表示为单个潜在令牌(动作原子)。通过自底向上的表示学习,由这些动作原子组成的时间模式自然出现(动作基元),这些模式捕获了可重复的、语义化的身体运动片段的有意义时间跨度。A4Mer通过在其各自的潜在空间中进行掩码令牌预测的统一预训练任务来实现这一点。我们还引入了动作基元数据集(AMD),这是一个大规模的多视角人类行为视频数据集,具有完整的SMPL注释。我们引入了一种新颖的相机使用方式,将其安装在脚上,以在频繁且严重的身体遮挡情况下实现逐帧注释。实验结果证明了A4Mer在提取有意义的动作基元方面的有效性,这些基元显著有益于人类行为建模任务,包括动作识别、运动预测和运动插值。

英文摘要:

Effective human behavior modeling requires a representation of the human body movement that capitalizes on its compositionality. We propose a hierarchical representation consisting of Action Atoms that capture the atomic joint movements and Action Motifs that are formed by their temporal compositions and encode similar body movements found across different overall human actions. We derive A4Mer, a nested latent Transformer to learn this hierarchical representation from human pose data in a fully self-supervised manner. A4Mer splits a 3D pose sequence into variable-length segments and represents each segment as a single latent token (Action Atoms). Through bottom-up representation learning, temporal patterns composed of these Action Atoms, which capture meaningful temporal spans of reusable, semantic segments of body movements, naturally emerge (Action Motifs). A4Mer achieves this with a unified pretext task of masked token prediction in their respective latent spaces. We also introduce Action Motif Dataset (AMD), a large-scale dataset of multi-view human behavior videos with full SMPL annotations. We introduce a novel use of cameras by mounting them on the feet to achieve their frame-wise annotations despite frequent and heavy body occlusions. Experimental results demonstrate the effectiveness of A4Mer for extracting meaningful Action Motifs, which significantly benefit human behavior modeling tasks including action recognition, motion prediction, and motion interpolation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑