MADiff：面向第一视角视频手部轨迹预测的运动感知Mamba扩散模型
MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videos
AI总结：
针对第一视角视频手部轨迹预测受相机自运动干扰、缺少可供性标签的难题，提出运动感知Mamba扩散模型MADiff，在五个公开数据集上达到与SOTA相当的预测精度且可实时运行，将开源代码与预训练模型。
摘要：
通过第一视角视频理解人类意图与动作是实现具身人工智能的重要环节。作为第一视角视觉技术的分支，手部轨迹预测在理解人类运动模式方面发挥着关键作用，可为扩展现实、机器人操作等下游任务提供支撑。然而，仅依靠第一视角视频时，捕捉符合合理时间因果关系的高层人类意图存在挑战，且相机自运动干扰、缺少可显式引导手部路径点分布优化的可供性标签会进一步加剧这一难度。本研究提出名为MADiff的新型手部轨迹预测方法，基于扩散模型预测未来手部路径点。该方法在潜空间设计的去噪操作由所提的运动感知Mamba实现，其中整合了相机佩戴者的自运动以完成运动驱动选择性扫描（MDSS）。为在无显式可供性监督的条件下识别手部与场景的关联，研究利用融合视觉与语言特征的基础模型，从视频片段中捕捉高层语义。研究在五个公开数据集上结合现有评估指标与新增评估指标开展了全面实验，结果表明MADiff预测的手部轨迹合理性与当前最优基线模型相当，且可实现实时性能。我们将在项目页面https://irmvlab.github.io/madiff.github.io/ 公开MADiff的代码与预训练模型。
英文摘要：
Understanding human intentions and actions through egocentric videos is important on the path to embodied artificial intelligence. As a branch of egocentric vision techniques, hand trajectory prediction plays a vital role in comprehending human motion patterns, benefiting downstream tasks in extended reality and robot manipulation. However, capturing high-level human intentions consistent with reasonable temporal causality is challenging when only egocentric videos are available. This difficulty is exacerbated under camera egomotion interference and the absence of affordance labels to explicitly guide the optimization of hand waypoint distribution. In this work, we propose a novel hand trajectory prediction method dubbed MADiff, which forecasts future hand waypoints with diffusion models. The devised denoising operation in the latent space is achieved by our proposed motion-aware Mamba, where the camera wearer's egomotion is integrated to achieve motion-driven selective scan (MDSS). To discern the relationship between hands and scenarios without explicit affordance supervision, we leverage a foundation model that fuses visual and language features to capture high-level semantics from video clips. Comprehensive experiments conducted on five public datasets with the existing and our proposed new evaluation metrics demonstrate that MADiff predicts comparably reasonable hand trajectories compared to the state-of-the-art baselines, and achieves real-time performance. We will release our code and pretrained models of MADiff at the project page: https://irmvlab.github.io/madiff.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
