发表机构
ACE Robotics; Nanyang Technological University; The Chinese University of Hong Kong; Shanghai Jiao Tong University（ACE机器人公司; 南洋理工大学; 香港中文大学; 上海交通大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
DreamHand是复用视频扩散模型的离线片段级框架，通过确定性干净潜在编码器与双向时空解码器恢复带度量位置的连续双手轨迹，在五个第一人称视角基准测试中实现最佳性能，为机器人操作数据提供可扩展路径。
AI 中文摘要
第一人称视角视频为具身AI提供了可扩展的操作数据，但由于严重的物体遮挡和频繁的视线外间隙，恢复度量标准的3D手部轨迹仍然具有挑战性。现有的单帧和窗口时间回归器在手部短暂离开画面时会失效，而近期的视频扩散模型（VDMs）依赖于繁重的随机多步采样作为像素空间渲染器。我们转而将VDM重新用作确定性几何编码器：对干净潜在空间的单次前向传播可暴露当前观测之外的场景内容，包括被遮挡和视线外的手部。我们提出DreamHand，这是一种离线片段级框架，通过确定性干净潜在编码器提取特征，并使用双向时空解码器对其进行解码。DreamHand可恢复带度量标准位置的连续双手轨迹，无需外部检测器；而基于射线的相机求解器支持第二种配置，无需测试时的相机内参。在五个第一人称视角基准测试中，DreamHand达到了新的最佳性能，在遮挡严重的ARCTIC上将MPJPE-p降低了30%，在HOT3D上降低了40%；当评估中纳入视线外的手部时，这些增益达到46%-61%，为从日常人类视频到机器人操作数据提供了可扩展的路径。
英文摘要
Egocentric video offers scalable manipulation data for embodied AI, yet recovering metric 3D hand trajectories remains challenging due to severe object occlusion and frequent out-of-sight gaps. Existing single-frame and windowed temporal regressors fail when a hand shortly leaves the frame, while recent video diffusion models (VDMs) rely on heavy, stochastic multi-step sampling as pixel-space renderers. We instead repurpose VDM into a deterministic geometry encoder. A single forward pass over the clean latent exposes scene content beyond current observations, including occluded and out-of-sight hands. We introduce ACE-Ego-Hand, an offline clip-level framework that extracts features via a Deterministic Clean-Latent Encoder and decodes them with a Bidirectional Spatiotemporal Decoder. ACE-Ego-Hand recovers continuous bimanual trajectories with metric placement and no external detector, while a Ray-Based Camera Solver supports a second configuration that requires no test-time camera intrinsics. Across five egocentric benchmarks, ACE-Ego-Hand sets a new state of the art, cutting MPJPE-p by 30% on occlusion-heavy ARCTIC and 40% on HOT3D. These gains reach 46%-61% once out-of-sight hands are included in the evaluation, offering a scalable path from everyday human video to robot manipulation data.
CommentsProject Page: https://ggxxii.github.io/ace-ego-hand
