掩码运动建模的提示对比学习：迈向通用的3D动作表示学习
Prompted Contrast with Masked Motion Modeling: Towards Versatile 3D Action Representation Learning
AI总结：
本文提出PCM³方法，通过互惠互利地融合对比学习与掩码运动建模，并引入双提示多任务预训练策略，显著提升了3D骨架动作表示在多种下游任务上的泛化能力。
摘要：
自监督学习已被证明对基于骨架的人体动作理解有效，这是一个重要但具有挑战性的课题。以往的工作主要依赖对比学习或掩码运动建模范式来建模骨架关系。然而，这些方法无法有效且同时地处理序列级和关节级的表示学习。因此，学习到的表示无法泛化到不同的下游任务。此外，以朴素的方式结合这两种范式无法发挥它们之间的协同作用，并可能导致训练中的干扰。为了解决这些问题，我们提出了Prompted Contrast with Masked Motion Modeling，即PCM$^{\ m 3}$，用于通用的3D动作表示学习。我们的方法以互惠互利的方式整合了对比学习和掩码预测任务，这大幅提升了对各种下游任务的泛化能力。具体而言，掩码预测为对比学习提供了新颖的训练视图，而对比学习反过来用高层语义信息指导掩码预测训练。此外，我们提出了一种双提示多任务预训练策略，通过减少学习两个不同前置任务所造成的干扰，进一步改善了模型表示。在三个大规模数据集上对五个下游任务进行了广泛实验，证明了PCM$^{\ m 3}$与最先进的工作相比具有优越的泛化能力。我们的项目公开可用：https://jhang2020.github.io/Projects/PCM3/PCM3.html。
英文摘要：
Self-supervised learning has proved effective for skeleton-based human action understanding, which is an important yet challenging topic. Previous works mainly rely on contrastive learning or masked motion modeling paradigm to model the skeleton relations. However, the sequence-level and joint-level representation learning cannot be effectively and simultaneously handled by these methods. As a result, the learned representations fail to generalize to different downstream tasks. Moreover, combining these two paradigms in a naive manner leaves the synergy between them untapped and can lead to interference in training. To address these problems, we propose Prompted Contrast with Masked Motion Modeling, PCM$^{\rm 3}$, for versatile 3D action representation learning. Our method integrates the contrastive learning and masked prediction tasks in a mutually beneficial manner, which substantially boosts the generalization capacity for various downstream tasks. Specifically, masked prediction provides novel training views for contrastive learning, which in turn guides the masked prediction training with high-level semantic information. Moreover, we propose a dual-prompted multi-task pretraining strategy, which further improves model representations by reducing the interference caused by learning the two different pretext tasks. Extensive experiments on five downstream tasks under three large-scale datasets are conducted, demonstrating the superior generalization capacity of PCM$^{\rm 3}$ compared to the state-of-the-art works. Our project is publicly available at: https://jhang2020.github.io/Projects/PCM3/PCM3.html .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
