arXiv 2511.128782026-05-11cs.CVcs.RO

Uni-Hand：面向亲身体验视角的通用手部运动预测

Uni-Hand: Universal Hand Motion Forecasting in Egocentric Views

IRMV Lab, Shanghai Jiao Tong University（上海交通大学IMV实验室）；Meta Reality Labs（Meta现实实验室）；Department of Electronic Engineering, Shanghai Jiao Tong University（上海交通大学电子工程系）；China University of Mining and Technology（中国矿业大学）；College of Intelligence Science and Technology, National University of Defense Technology（国防科技大学智能科学与技术学院）

Junyi Ma, Wentao Bao, Jingyi Xu, Guanzhong Sun, Yu Zheng, Erhang Zhang, Xieyuanli Chen, Hesheng Wang

备注：: Accepted by T-PAMI 2026. Code and data: https://github.com/IRMVLab/UniHand

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Uni-Hand框架，通过多模态输入、多维多目标预测和多任务赋能，实现手部在2D和3D空间的精准预测，并引入目标指标预测手腕和指尖关节点，同时预测手-物体交互状态，提升下游任务性能。

摘要：

在亲身体验视角中预测人类手部运动对于增强现实和人机机器人策略转移等应用至关重要。最近，几种手轨迹预测（HTP）方法被开发出来以生成未来可能的手部路径点，但仍存在预测目标不足、固有模态差距、手-头部运动交织以及在下游任务中的验证有限等问题。为了解决这些限制，我们提出了一种通用手部运动预测框架，考虑多模态输入、多维和多目标预测模式以及为下游应用设计的多任务赋能。我们通过视觉-语言融合、全局上下文整合和任务感知文本嵌入注入来和谐多种模态，以预测手部路径点在2D和3D空间中的位置。我们提出了一种新的双分支扩散模型，同时预测人类头部和手部运动，捕捉其在亲身体验视觉中的运动协同性。通过引入目标指标，预测模型可以预测手腕或手指的特定关节路径点，而不仅仅是广泛研究的手中心点。此外，我们使Uni-Hand能够进一步预测手-物体交互状态（接触/分离），以更好地促进下游任务。作为首次在文献中将下游任务评估纳入的论文，我们构建了新的基准以评估手部运动预测算法在现实中的适用性。在多个公开可用数据集和我们新提出的基准上的实验结果表明，Uni-Hand在多维和多目标手部运动预测中实现了最先进的性能。在多个下游任务中的广泛验证也展示了其出色的机器人操作策略转移能力，以及在动作预期/识别中的有效特征增强。

英文摘要：

Forecasting how human hands move in egocentric views is critical for applications like augmented reality and human-robot policy transfer. Recently, several hand trajectory prediction (HTP) methods have been developed to generate future possible hand waypoints, which still suffer from insufficient prediction targets, inherent modality gaps, entangled hand-head motion, and limited validation in downstream tasks. To address these limitations, we present a universal hand motion forecasting framework considering multi-modal input, multi-dimensional and multi-target prediction patterns, and multi-task affordances for downstream applications. We harmonize multiple modalities by vision-language fusion, global context incorporation, and task-aware text embedding injection, to forecast hand waypoints in both 2D and 3D spaces. A novel dual-branch diffusion is proposed to concurrently predict human head and hand movements, capturing their motion synergy in egocentric vision. By introducing target indicators, the prediction model can forecast the specific joint waypoints of the wrist or the fingers, besides the widely studied hand center points. In addition, we enable Uni-Hand to additionally predict hand-object interaction states (contact/separation) to facilitate downstream tasks better. As the first work to incorporate downstream task evaluation in the literature, we build novel benchmarks to assess the real-world applicability of hand motion forecasting algorithms. The experimental results on multiple publicly available datasets and our newly proposed benchmarks demonstrate that Uni-Hand achieves the state-of-the-art performance in multi-dimensional and multi-target hand motion forecasting. Extensive validation in multiple downstream tasks also presents its impressive human-robot policy transfer to enable robotic manipulation, and effective feature enhancement for action anticipation/recognition.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。