arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.09646cs.CVcs.RO
收藏

Hamba: 基于图引导双扫描Mamba的单视角3D手形重建

Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mamba

Carnegie Mellon University(卡内基梅隆大学)

Haoye Dong, Aviral Chharia, Wenbo Gou, Francisco Vicente Carrasco, Fernando De la Torre

AI总结:

Hamba通过图引导双扫描Mamba框架,高效学习关节空间关系,实现更准确的3D手形重建。

摘要:

从单个RGB图像进行3D手形重建具有挑战性,由于关节运动、自遮挡以及与物体的相互作用。现有最先进的方法采用基于注意力的Transformer来学习3D手形和形状,但它们未能完全实现鲁棒和准确的性能,主要原因是不高效地建模关节之间的空间关系。为了解决这个问题,我们提出了一种新的图引导Mamba框架,称为Hamba,它连接了图学习和状态空间建模。我们的核心思想是将Mamba的扫描重新公式化为图引导的双向扫描,用于使用少量有效标记的3D重建。这使我们能够高效地学习关节之间的空间关系以提高重建性能。具体来说,我们设计了一个图引导的状态空间(GSS)块,该块学习关节的图结构关系和空间序列,并使用比基于注意力的方法少88.5%的标记。此外,我们通过融合模块整合状态空间特征和全局特征。通过利用GSS块和融合模块,Hamba有效地利用了图引导的状态空间特征,并共同考虑了全局和局部特征以提高性能。在多个基准和真实场景测试中,Hamba显著优于现有最先进的方法,在FreiHAND上实现了PA-MPVPE为5.3mm和F@15mm为0.992的性能。在本文接受时,Hamba在3D手形重建的两个竞赛排行榜上排名第一。项目网站:https://humansensinglab.github.io/Hamba/

英文摘要:

3D Hand reconstruction from a single RGB image is challenging due to the articulated motion, self-occlusion, and interaction with objects. Existing SOTA methods employ attention-based transformers to learn the 3D hand pose and shape, yet they do not fully achieve robust and accurate performance, primarily due to inefficiently modeling spatial relations between joints. To address this problem, we propose a novel graph-guided Mamba framework, named Hamba, which bridges graph learning and state space modeling. Our core idea is to reformulate Mamba's scanning into graph-guided bidirectional scanning for 3D reconstruction using a few effective tokens. This enables us to efficiently learn the spatial relationships between joints for improving reconstruction performance. Specifically, we design a Graph-guided State Space (GSS) block that learns the graph-structured relations and spatial sequences of joints and uses 88.5% fewer tokens than attention-based methods. Additionally, we integrate the state space features and the global features using a fusion module. By utilizing the GSS block and the fusion module, Hamba effectively leverages the graph-guided state space features and jointly considers global and local features to improve performance. Experiments on several benchmarks and in-the-wild tests demonstrate that Hamba significantly outperforms existing SOTAs, achieving the PA-MPVPE of 5.3mm and F@15mm of 0.992 on FreiHAND. At the time of this paper's acceptance, Hamba holds the top position, Rank 1 in two Competition Leaderboards on 3D hand reconstruction. Project Website: https://humansensinglab.github.io/Hamba/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑