arXiv 2504.017642025-04-02cs.CVcs.AI

基于上下文表征学习的双流Transformer-GCN模型用于单目3D人体姿态估计

Dual-stream Transformer-GCN Model with Contextualized Representations Learning for Monocular 3D Human Pose Estimation

Northeastern University（东北大学）；College of Sciences, Northeastern University（东北大学理学院）；Key Laboratory of Differential Equations and Their Applications, Northeastern University（东北大学微分方程及应用重点实验室）；Liaoning Provincial Department of Education（辽宁省教育厅）；King’s College London（伦敦国王学院）；Shenyang sport University（沈阳体育学院）

Mingrui Ye, Lianping Yang, Hegui Zhu, Zenghao Zheng, Xin Wang, Yantao Lo

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出基于上下文表征学习的Transformer-GCN双流模型，通过掩码2D姿态特征与自蒸馏预训练增强时空建模，在单目3D人体姿态估计任务上实现SOTA性能与强泛化能力。

摘要：

本文介绍了一种使用基于上下文表征学习的Transformer-GCN双流模型进行单目3D人体姿态估计的新方法。单目3D人体姿态估计面临深度模糊、3D标注训练数据有限、建模不平衡以及模型泛化受限等挑战。为了解决这些局限性，我们的工作引入了一种基于上下文表征学习的开创性运动预训练方法。具体而言，该方法包括掩蔽2D姿态特征，并利用Transformer-GCN双流模型通过自蒸馏设置来学习高维表征。通过专注于上下文表征学习和时空建模，该方法增强了模型理解姿态之间时空关系的能力，从而实现了卓越的泛化能力。此外，利用Transformer-GCN双流模型，该方法有效平衡了视频姿态估计中的全局和局部交互。该模型自适应地整合来自Transformer和GCN双流的信息，其中GCN流有效学习相邻关键点和帧之间的局部关系，而Transformer流捕获全面的全局空间和时间特征。我们的模型在两个基准数据集上取得了最先进的性能，在Human3.6M上的MPJPE为38.0mm，P-MPJPE为31.9mm，在MPI-INF-3DHP上的MPJPE为15.9mm。此外，在公开数据集和野外视频上的视觉实验证明了该方法的鲁棒性和泛化能力。

英文摘要：

This paper introduces a novel approach to monocular 3D human pose estimation using contextualized representation learning with the Transformer-GCN dual-stream model. Monocular 3D human pose estimation is challenged by depth ambiguity, limited 3D-labeled training data, imbalanced modeling, and restricted model generalization. To address these limitations, our work introduces a groundbreaking motion pre-training method based on contextualized representation learning. Specifically, our method involves masking 2D pose features and utilizing a Transformer-GCN dual-stream model to learn high-dimensional representations through a self-distillation setup. By focusing on contextualized representation learning and spatial-temporal modeling, our approach enhances the model's ability to understand spatial-temporal relationships between postures, resulting in superior generalization. Furthermore, leveraging the Transformer-GCN dual-stream model, our approach effectively balances global and local interactions in video pose estimation. The model adaptively integrates information from both the Transformer and GCN streams, where the GCN stream effectively learns local relationships between adjacent key points and frames, while the Transformer stream captures comprehensive global spatial and temporal features. Our model achieves state-of-the-art performance on two benchmark datasets, with an MPJPE of 38.0mm and P-MPJPE of 31.9mm on Human3.6M, and an MPJPE of 15.9mm on MPI-INF-3DHP. Furthermore, visual experiments on public datasets and in-the-wild videos demonstrate the robustness and generalization capabilities of our approach.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。