AssemblyHands-X：建模3D手-身体协调以理解双手人类活动
AssemblyHands-X: Modeling 3D Hand-Body Coordination for Understanding Bimanual Human Activities
AI总结：
本文提出无标记3D手-身体基准AssemblyHands-X，结合多视角三角测量与SMPL-X拟合生成标注，并证明联合建模手身姿态可提升双手动作识别。
摘要：
双手人类活动本质上涉及双手与身体的协调运动。然而，由于缺乏合适的数据集，这种协调在活动理解中的影响尚未得到系统评估。此类评估需要手部和身体的运动学层级标注（例如3D姿态），但现有3D活动数据集通常只标注手部或身体姿态。另一类工作采用基于标记的动作捕捉来提供全身姿态，但物理标记会引入视觉伪影，从而限制模型向自然、无标记视频的泛化。为解决这些局限，我们提出AssemblyHands-X，这是首个面向双手活动的无标记3D手-身体基准，旨在研究手-身体协调对动作识别的影响。我们首先构建一条从同步多视角视频进行3D姿态标注的流水线。该方法将多视角三角测量与SMPL-X网格拟合相结合，实现手部和上半身可靠的3D配准。随后，我们在基于图卷积或时空注意力的近期动作识别模型中，验证不同输入表示（例如视频、手部姿态、身体姿态或手-身体姿态）。大量实验表明，基于姿态的动作推理比视频基线更高效、更准确。此外，与单独使用手部或上半身相比，联合建模手部和身体线索可提升动作识别性能，凸显了建模相互依赖的手-身体动态对于整体理解双手活动的重要性。
英文摘要：
Bimanual human activities inherently involve coordinated movements of both hands and body. However, the impact of this coordination in activity understanding has not been systematically evaluated due to the lack of suitable datasets. Such evaluation demands kinematic-level annotations (e.g., 3D pose) for the hands and body, yet existing 3D activity datasets typically annotate either hand or body pose. Another line of work employs marker-based motion capture to provide full-body pose, but the physical markers introduce visual artifacts, thereby limiting models' generalization to natural, markerless videos. To address these limitations, we present AssemblyHands-X, the first markerless 3D hand-body benchmark for bimanual activities, designed to study the effect of hand-body coordination for action recognition. We begin by constructing a pipeline for 3D pose annotation from synchronized multi-view videos. Our approach combines multi-view triangulation with SMPL-X mesh fitting, yielding reliable 3D registration of hands and upper body. We then validate different input representations (e.g., video, hand pose, body pose, or hand-body pose) across recent action recognition models based on graph convolution or spatio-temporal attention. Our extensive experiments show that pose-based action inference is more efficient and accurate than video baselines. Moreover, joint modeling of hand and body cues improves action recognition over using hands or upper body alone, highlighting the importance of modeling interdependent hand-body dynamics for a holistic understanding of bimanual activities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
