arXiv 2504.073752025-11-14cs.CV

用于多模态3D手部轨迹预测的新型扩散模型

Novel Diffusion Models for Multimodal 3D Hand Trajectory Prediction

IRMV Lab, the Department of Automation, Shanghai Jiao Tong University（IRMV实验室，自动化系，上海交通大学）；Meta Reality Labs（Meta现实实验室）；the Department of Electronic Engineering, Shanghai Jiao Tong University（电子工程系，上海交通大学）；the School of Information and Control Engineering, China University of Mining and Technology（信息与控制工程学院，中国矿业大学）；the College of Intelligence Science and Technology, National University of Defense Technology（智能科学与技术学院，国防科技大学）

Junyi Ma, Wentao Bao, Jingyi Xu, Guanzhong Sun, Xieyuanli Chen, Hesheng Wang

备注：: Accepted to IROS 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有3D手部轨迹预测方法缺乏多模态感知、忽略手部运动与相机自运动协同的问题，提出MMTwin孪生扩散模型，融合多模态输入实现自运动与手部轨迹同步预测，性能更优且泛化性良好。

摘要：

手部运动预测对于理解人类意图、衔接人类动作与机器人操作的动作空间至关重要。现有手部轨迹预测（HTP）方法以过往第一视角观测为条件，预测3D空间中未来的手部路径点，但这类模型仅适配2D第一视角视频输入，缺乏对来自2D和3D观测的多模态环境信息的感知，阻碍了3D手部轨迹预测性能的进一步提升。此外，这些模型忽略了手部运动与头显相机自运动之间的协同作用，要么孤立预测手部轨迹，要么仅从过往帧编码自运动。为解决上述局限，我们提出了用于多模态3D手部轨迹预测的新型扩散模型MMTwin。MMTwin可接收包含2D RGB图像、3D点云、过往手部路径点及文本提示的多模态信息作为输入；同时模型集成了作为孪生结构的两个潜在扩散模型——自运动扩散模型与HTP扩散模型，可同时预测相机自运动与未来手部轨迹。我们提出了新型混合Mamba-Transformer模块作为HTP扩散模型的去噪模型，以更好地融合多模态特征。在三个公开数据集及我们自采数据上的实验结果表明，与当前最优基线方法相比，所提出的MMTwin可预测合理的未来3D手部轨迹，且在未见过的环境中具备良好的泛化性。代码与预训练模型已发布于https://github.com/IRMVLab/MMTwin。

英文摘要：

Predicting hand motion is critical for understanding human intentions and bridging the action space between human movements and robot manipulations. Existing hand trajectory prediction (HTP) methods forecast the future hand waypoints in 3D space conditioned on past egocentric observations. However, such models are only designed to accommodate 2D egocentric video inputs. There is a lack of awareness of multimodal environmental information from both 2D and 3D observations, hindering the further improvement of 3D HTP performance. In addition, these models overlook the synergy between hand movements and headset camera egomotion, either predicting hand trajectories in isolation or encoding egomotion only from past frames. To address these limitations, we propose novel diffusion models (MMTwin) for multimodal 3D hand trajectory prediction. MMTwin is designed to absorb multimodal information as input encompassing 2D RGB images, 3D point clouds, past hand waypoints, and text prompt. Besides, two latent diffusion models, the egomotion diffusion and the HTP diffusion as twins, are integrated into MMTwin to predict camera egomotion and future hand trajectories concurrently. We propose a novel hybrid Mamba-Transformer module as the denoising model of the HTP diffusion to better fuse multimodal features. The experimental results on three publicly available datasets and our self-recorded data demonstrate that our proposed MMTwin can predict plausible future 3D hand trajectories compared to the state-of-the-art baselines, and generalizes well to unseen environments. The code and pretrained models have been released at https://github.com/IRMVLab/MMTwin.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。