在手-物交互场景中通过聚合先验知识引导的图内与图间信息进行3D手部重建
3D Hand Reconstruction via Aggregating Intra and Inter Graphs Guided by Prior Knowledge for Hand-Object Interaction Scenario
AI总结:
针对手-物交互中的遮挡与参数回归难题,提出融合MANO先验、2D关节直接回归及顶点-关节互图注意力的3D手部重建网络,在HO3DV2和Dex-YCB上优于单一范式方法。
摘要:
近年来,3D手部重建在人机协作中受到越来越多的关注,尤其是在手-物交互场景中。然而,由于交互导致严重的手部遮挡,这仍然是一项巨大挑战,其中包括准确性与物理合理性之间的平衡、模型参数的高度非线性映射以及遮挡特征增强。为克服这些问题,我们提出了一种3D手部重建网络,结合基于模型方法和无模型方法的优势,以在手-物交互场景中平衡准确性与物理合理性。首先,我们提出了一个新颖的、直接从2D关节点回归MANO姿态参数的模块,避免了从抽象图像特征进行高度非线性映射的过程,并且不再依赖精确的3D关节点。此外,我们进一步提出了一种由MANO引导的顶点-关节点互图注意力模型,用于联合优化手部网格和关节点;该模型对顶点-顶点以及关节点-关节点之间的依赖关系进行建模,并捕获顶点-关节点之间的相关性,从而分别聚合图内和图间的节点特征。实验结果表明,我们的方法在近期的基准数据集HO3DV2和Dex-YCB上取得了具有竞争力的性能,并且优于所有仅采用基于模型方法和无模型方法的方法。
英文摘要:
Recently, 3D hand reconstruction has gained more attention in human-computer cooperation, especially for hand-object interaction scenario. However, it still remains huge challenge due to severe hand-occlusion caused by interaction, which contain the balance of accuracy and physical plausibility, highly nonlinear mapping of model parameters and occlusion feature enhancement. To overcome these issues, we propose a 3D hand reconstruction network combining the benefits of model-based and model-free approaches to balance accuracy and physical plausibility for hand-object interaction scenario. Firstly, we present a novel MANO pose parameters regression module from 2D joints directly, which avoids the process of highly nonlinear mapping from abstract image feature and no longer depends on accurate 3D joints. Moreover, we further propose a vertex-joint mutual graph-attention model guided by MANO to jointly refine hand meshes and joints, which model the dependencies of vertex-vertex and joint-joint and capture the correlation of vertex-joint for aggregating intra-graph and inter-graph node features respectively. The experimental results demonstrate that our method achieves a competitive performance on recently benchmark datasets HO3DV2 and Dex-YCB, and outperforms all only model-base approaches and model-free approaches.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。