可控的第一人称视角视频生成：基于遮挡感知的稀疏3D手部关节点
Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints
AI总结：
提出利用稀疏3D手部关节点作为显式控制信号，通过遮挡感知特征提取和3D加权机制，实现高保真、3D一致的第一人称手物交互视频生成。
摘要：
复杂手物交互的可控视频生成是构建视觉世界模型的关键步骤。然而，现有方法在生成视频中难以实现精细且3D一致的手部关节运动。通过依赖密集2D轨迹或隐式姿态表示，它们将关键的几何结构压缩为空间模糊的信号，导致在第一人称遮挡下出现严重的运动不一致和幻觉伪影。为解决这一问题，我们提出利用稀疏3D手部关节点作为显式控制信号，具有三个关键优势：显式几何以解决遮挡、用于交互式编辑的直观接口、以及跨具身泛化到机器人手。在此基础上，我们的高效控制模块通过惩罚来自隐藏关节的不可靠视觉特征，从源参考帧中提取遮挡感知特征，并采用基于3D的加权机制来处理运动传播过程中动态遮挡的目标关节。同时，它直接将3D几何嵌入注入潜在空间以强制结构一致性。为促进鲁棒训练和评估，我们开发了一个自动标注流程，生成了100万个高质量的第一人称视频片段，并配以精确的手部轨迹。实验表明，我们的方法优于最先进的基线，生成了具有真实手物交互的高保真第一人称视频。
英文摘要：
Controllable video generation for complex hand-object interactions is a critical step toward building visual world models. However, existing methods often struggle to achieve fine-grained, 3D-consistent hand articulation in generated videos. By relying on dense 2D trajectories or implicit pose representations, they collapse crucial geometric structures into spatially ambiguous signals, leading to severe motion inconsistencies and hallucinated artifacts under egocentric occlusions. To address this, we propose leveraging sparse 3D hand joints as explicit control signals with three key advantages: explicit geometry to resolve occlusions, an intuitive interface for interactive editing, and cross-embodiment generalization to robotic hands. Built upon this, our efficient control module extracts occlusion-aware features from the source reference frame by penalizing unreliable visual features from hidden joints, and employs a 3D-based weighting mechanism to handle dynamically occluded target joints during motion propagation. Meanwhile, it directly injects 3D geometric embeddings into the latent space to enforce structural consistency. To facilitate robust training and evaluation, we develop an automated annotation pipeline, yielding 1M high-quality egocentric video clips paired with precise hand trajectories. Experiments demonstrate that our approach outperforms state-of-the-art baselines, generating high-fidelity egocentric videos with realistic hand-object interactions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
