ESTHER：野外环境下以自我为中心的立体手部估计与重建

中文摘要 AI 辅助

人类灵巧性由两只眼睛观察两只手来引导：双目视觉提供了精细操作所需的度量三维结构。因此，以自我为中心的立体视觉是机器人、增强现实和虚拟现实的天然感知接口——然而，从这一信号进行度量三维手部重建，既没有端到端模型，也没有野外基准。我们提出了ESTHER，一个其立体几何、时间推理和输出表示专为可穿戴以自我为中心的立体视觉设计的模型。它通过校准标注流程生成的伪标签进行训练，并进而构建了我们的基准ESTHER3D，这是一个以自我为中心的立体手部数据集，将大规模野外训练集（包含模型生成的标签）与运动捕捉测试集（包含真实度量真值）配对。实验表明，该模型达到了最先进的准确性、优越的外部泛化能力，并且对真实以自我为中心捕捉中出现的缺失视角、丢帧、光照和运动模糊极端情况具有鲁棒性，而这些情况会使现有方法失效。这种鲁棒性超越了简单的优雅降级：立体引导教会模型将表观手部尺度与度量深度绑定，因此它不仅能在极少微调下适应不同的立体设备和模态，更引人注目的是，即使在退化为单一单目视图后，它仍能保持真实的度量尺度。

英文摘要

Human dexterity is guided by two eyes watching two hands: binocular vision supplies the metric 3D structure that fine-grained manipulation consumes. Egocentric stereo is therefore the natural perceptual interface for robots, AR, and VR-yet metric 3D hand reconstruction from this very signal still has neither an end-to-end model nor an in-the-wild benchmark. We propose ESTHER, a model whose stereo geometry, temporal reasoning, and output representation are designed for wearable egocentric stereo. It is trained on pseudo-labels from a calibrated labeling pipeline and in turn assembles our benchmark ESTHER3D, an egocentric stereo hand dataset pairing a large in-the-wild training set of model-generated labels with a motion capture test set of true metric ground truth. Experiments show state-of-the-art accu?racy, superior external generalization, and robustness to the missing views, dropped frames, and lighting and motion blur extremes of real egocentric capture that break existing meth?ods. This robustness runs deeper than graceful degradation: stereo guidance teaches the model to bind apparent hand scale to metric depth, so it not only adapts to different stereo rigs and modalities with minimal fine-tuning, but more strikingly preserves true metric scale even after collapsing to a single monocular view.

发表机构

• City University of Hong Kong（香港城市大学）

机构由 AI 辅助整理，请以论文原文为准。