EggHand：一种用于第一人称手姿态预测的多模态基础模型
EggHand: A Multimodal Foundation Model for Egocentric Hand Pose Forecasting
AI总结：
本文提出EggHand模型，通过结合视觉-语言-动作模型的动作解码器和第一人称视频-文本编码器，实现对第一人称视频中手姿态序列的预测，提升了在剧烈视角变化下的鲁棒性和可控性。
摘要：
从第一人称视频预测未来3D手姿态序列对于理解人类意图和实现AR/VR辅助以及人机交互等应用至关重要。然而，这一任务极具挑战性，因为第一人称手运动由复杂的人类意图驱动，表现出高度灵活的关节运动，并且在由于自我运动引起的剧烈视角变化下被观察到。在本文中，我们引入了EggHand，一种基于基础模型的框架，用于第一人称手姿态预测，该框架统一了多模态语义推理与动态运动建模。我们的方法结合了来自视觉-语言-动作（VLA）模型的动作解码器，该解码器捕捉手运动的结构时间动态，以及一个提供从大规模第一人称视频中学习到的视角感知上下文信息的第一人称视频-文本编码器。这些组件共同克服了在自我运动下通用视觉编码器的脆弱性，并实现了在不依赖身体姿态或外部跟踪的情况下对运动、上下文和高级意图的联合推理。在EgoExo4D数据集上的实验表明，EggHand在预测准确性方面达到了新的状态，保持了在严重自我运动下的鲁棒性，并进一步通过基于语言的任务提示实现了可控的预测。项目页面：https://jyoun9.github.io/EggHand
英文摘要：
Forecasting future 3D hand pose sequences from egocentric video is essential for understanding human intention and enabling embodied applications such as AR/VR assistance and human-robot interaction. However, this task remains a highly challenging problem because egocentric hand motion is driven by complex human intent, exhibits highly dexterous articulations, and is observed under drastic viewpoint shifts induced by ego-motion. In this work, we introduce EggHand, a foundation-model-based framework for egocentric hand pose forecasting that unifies multimodal semantic reasoning with dynamic motion modeling. Our approach couples an action decoder from a Vision-Language-Action (VLA) model, which captures the structured temporal dynamics of hand motion, with an egocentric video-text encoder that provides viewpoint-aware contextual information learned from large-scale first-person video. Together, these components overcome the brittleness of generic visual encoders under ego-motion and enable joint reasoning over motion, context, and high-level intent-without relying on body pose or external tracking. Experiments on the EgoExo4D dataset show that EggHand sets a new state of the art in forecasting accuracy, remains robust under severe ego-motion, and further enables controllable prediction via language-based task prompts. Project page: https://jyoun9.github.io/EggHand
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
