arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.29936cs.CV
收藏

DewHand:从第一视角视频直接重建度量世界空间的手部运动

DewHand: Direct Metric World-Space Hand Motion Reconstruction from Egocentric Videos

The University of Tokyo(东京大学);National Institute of Informatics(信息学研究科)

Naru Suzuki, Satoshi Ikehata, Takehiko Ohkawa, Yoichi Sato, Shuhei Kurita

AI总结:

DewHand是一种从第一视角视频直接重建世界空间手部与相机运动的统一模型,通过单阶段联合预测、结合VGGT-Ω特征与流匹配生成,在多个数据集上优于现有最优方法。

摘要:

从第一视角视频中恢复度量世界空间的手部运动,有助于理解人类交互并将其迁移到具身智能体。该任务面临三大挑战:多阶段流水线会传播独立估计的手部与相机运动产生的误差;单目观测导致物理尺度模糊;手部遮挡或超出视野会产生多种合理的运动轨迹。本文提出DewHand,一种用于直接联合重建世界空间手部与相机运动的统一模型。为避免多阶段估计的误差累积,DewHand在单阶段流水线内的共享世界坐标系中联合预测手部与相机运动。为解决度量尺度模糊问题,它结合来自VGGT-Ω的场景级几何特征与度量空间监督,同时手部特定特征保留精细的关节结构。为处理手部遮挡与超出视野的情况,本文将重建建模为基于流匹配的条件运动生成。在相同架构下,这种生成式建模仅需两次采样步骤,在精度与时间平滑度上均优于直接回归方法。在HOT3D与ARCTIC数据集上的实验表明,DewHand相比现有最优方法提升了世界空间精度与时间一致性。在HOT3D数据集上,DewHand相对于StableHand的刚性对齐世界空间误差从32.9 mm降至28.4 mm;在ARCTIC数据集上,该误差从76.2 mm降至36.2 mm。此外,在三分之二的HOT3D片段中,DewHand能将相机尺度恢复至10%以内,而SLAM方法仅约四分之一的片段达到该精度。在使用不同设备采集的HoloAssist、H2O与TACO的零样本评估中,DewHand在每个数据集上均实现了最低的刚性对齐世界空间误差(WA-MPJPE)。

英文摘要:

Recovering metric world-space hand motion from egocentric video supports understanding human interactions and transferring them to embodied agents. Three challenges complicate this task: multi-stage pipelines propagate errors from independently estimated hand and camera motion; monocular observations leave physical scale ambiguous; and occlusion and out-of-view hands admit multiple plausible motion trajectories. We present DewHand, a unified model for direct joint reconstruction of world-space hand and camera motion. To avoid compounding errors from multi-stage estimation, DewHand jointly predicts hand and camera motion in a shared world coordinate system within a single-stage pipeline. To resolve metric-scale ambiguity, it combines scene-level geometric features from VGGT-$Ω$ with metric-space supervision, while hand-specific features preserve fine-grained articulation. To handle occlusion and out-of-view hands, we formulate reconstruction as conditional motion generation with flow matching. Under the same architecture, this generative formulation outperforms direct regression in both accuracy and temporal smoothness with only two sampling steps. Experiments on HOT3D and ARCTIC demonstrate improved world-space accuracy and temporal consistency over state-of-the-art methods. DewHand reduces rigid-aligned world-space error relative to StableHand from 32.9 to 28.4 mm on HOT3D and from 76.2 to 36.2 mm on ARCTIC. It also recovers camera scale within 10% on two thirds of HOT3D clips, versus about a quarter for SLAM. Moreover, in zero-shot evaluations on HoloAssist, H2O, and TACO, captured with different devices, DewHand achieves the lowest rigid-aligned world-space error (WA-MPJPE) on each dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑