学习语义动态与时空协作以实现视频中的人体姿态估计
Learning semantical dynamics and spatiotemporal collaboration for human pose estimation in video
AI总结：
针对现有视频人体姿态估计方法依赖局部像素相似性、忽略语义相关性且模态融合简单的问题，本文提出含多级语义运动编码器和空间-运动互学习模块的框架，在三个PoseTrack数据集上取得新SOTA。
摘要：
时间建模和时空协作是基于视频的人体姿态估计的关键技术。大多数最先进的方法采用光流或时间差分，在像素级别学习跨帧的局部视觉内容对应关系，以捕捉运动动态。然而，这种范式本质上依赖于局部像素间的相似性，忽略了帧间的语义相关性，且易受图像质量下降（如遮挡或模糊）的影响。此外，现有方法常通过简单的拼接或求和来组合运动和空间（外观）特征，导致在充分利用这些不同模态方面面临实际挑战。在本文中，我们提出了一个新颖的框架，用于学习多级语义动态和密集时空协作，以实现多帧人体姿态估计。具体而言，我们首先设计了一个多级语义运动编码器，使用多掩码上下文和姿态重建策略。该策略通过逐步掩码（补丁）立方体和帧的特征，促使模型探索帧间的多粒度时空语义关系。我们进一步引入了空间-运动互学习模块，该模块密集传播和整合来自空间和运动特征的上下文信息，以增强模型的能力。大量实验表明，我们的方法在三个基准数据集PoseTrack2017、PoseTrack2018和PoseTrack21上取得了新的最先进结果。
英文摘要：
Temporal modeling and spatio-temporal collaboration are pivotal techniques for video-based human pose estimation. Most state-of-the-art methods adopt optical flow or temporal difference, learning local visual content correspondence across frames at the pixel level, to capture motion dynamics. However, such a paradigm essentially relies on localized pixel-to-pixel similarity, which neglects the semantical correlations among frames and is vulnerable to image quality degradations (e.g. occlusions or blur). Moreover, existing approaches often combine motion and spatial (appearance) features via simple concatenation or summation, leading to practical challenges in fully leveraging these distinct modalities. In this paper, we present a novel framework that learns multi-level semantical dynamics and dense spatio-temporal collaboration for multi-frame human pose estimation. Specifically, we first design a Multi-Level Semantic Motion Encoder using a multi-masked context and pose reconstruction strategy. This strategy stimulates the model to explore multi-granularity spatiotemporal semantic relationships among frames by progressively masking the features of (patch) cubes and frames. We further introduce a Spatial-Motion Mutual Learning module which densely propagates and consolidates context information from spatial and motion features to enhance the capability of the model. Extensive experiments demonstrate that our approach sets new state-of-the-art results on three benchmark datasets, PoseTrack2017, PoseTrack2018, and PoseTrack21.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
