发表机构
The University of Hong Kong; UC Berkeley（香港大学; 加州大学伯克利分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对视频中手部运动恢复因遮挡而不准确的问题，提出生成式框架JoHan，联合生成二维和三维姿态序列，利用二维线索和运动先验，显著提升精度、速度与运动平滑度。
AI 中文摘要
从视频中恢复忠实的三维手部运动仍然具有挑战性，因为频繁的遮挡和不完整的视觉观察使得逐帧姿态估计不可靠且时间上不一致。为了解决这个问题，我们提出了JoHan，一个统一的生成式框架，直接从视频序列中恢复手部运动，而不依赖于中间的单帧姿态预测。我们的模型从头开始训练，通过联合学习二维和三维局部手部姿态序列的时间动态以及跨表征对应关系，生成对齐的二维和三维局部手部姿态序列。生成的二维轨迹利用二维图像中的直接空间和时间线索来指导后续的生成式三维运动重建，而学习到的运动先验则促进时间一致性。它们学习到的二维-三维对应关系进一步使得能够恢复手部相对于相机的全局位置和方向。在具有挑战性的基准上的大量实验表明，在局部手部姿态和相机空间重建方面，准确性和速度均有显著提升。值得注意的是，我们的方法捕捉到了更好的手部运动动态，产生比先前方法明显更平滑的运动，同时保持了较高的逐帧姿态准确性。
英文摘要
Recovering faithful 3D hand motion from video remains challenging due to frequent occlusions and incomplete visual observations, which make frame-wise pose estimates unreliable and temporally inconsistent. To address this problem, we propose JoHan, a unified generative framework that recovers hand motion directly from video sequences without relying on intermediate per-frame pose predictions. Trained from scratch, our model jointly generates aligned 2D and 3D local hand pose sequences by learning their temporal dynamics and cross-representation correspondence. The generated 2D trajectories exploit direct spatial and temporal cues from the 2D images to guide the following generative 3D motion reconstruction, while the learned motion prior promotes temporal consistency. Their learned 2D-3D correspondence further enables recovery of the hand's global position and orientation relative to the camera. Extensive experiments on challenging benchmarks demonstrate significantly improved accuracy and speed in local hand-pose and camera-space reconstruction. Notably, our method captures much better hand-motion dynamics, producing significantly smoother motion than previous methods while maintaining high per-frame pose accuracy.
Comments20 pages, 6 figures
