3PoinTr: 从无约束人类视频中通过3D点轨迹学习操作
3PoinTr: From Human Videos to Robot Policies with 3D Point-Track Plans
AI总结：
提出3PoinTr方法，通过预测密集3D点轨迹从无约束人类视频中预训练样本高效的机器人操作策略，在真实和模拟任务中显著优于基线方法。
摘要：
从人类视频中学习操作策略可以大大减少对昂贵的机器人演示的需求，但现有方法通常需要限制性假设，如编排的人类动作、预定义关键点、手动注释或已知的抓取位置。我们提出3PoinTr，一种通过预测密集3D点轨迹从无约束人类视频中预训练样本高效机器人策略的方法。在无约束的人类演示视频中，人类可以自由地遵循他们认为合适的任何轨迹和操作策略，而不是编排他们的动作来模仿机器人。3PoinTr使用轻量级的可见性感知Transformer从人类视频中学习场景点应如何移动，然后训练一个闭环多任务机器人策略，以从这些预测的点轨迹中灵活提取与动作相关的先验知识。仅使用20个带动作标签的机器人演示，3PoinTr在真实世界任务上的平均成功率比最强的行为克隆和视频预训练基线高出25.0个百分点，在模拟中平均成功率高出29.6个百分点。针对性的消融实验支持关键设计选择，并证实了从无动作视频中学习的好处。我们进一步表明，3PoinTr的点轨迹预测Transformer通过保留对部分遮挡点的监督，优于强基线。项目页面：https://adamhung60.github.io/3PoinTr/。
英文摘要：
Human videos could reduce the reliance on expensive robot demonstrations, but extracting task-relevant features remains challenging. We introduce 3PoinTr, a method that provides robot policies with a global 3D point-track plan predicted from human videos. Given an initial point cloud, 3PoinTr predicts dense 3D point tracks spanning the full task and compresses them into a small set of tokens that remain fixed throughout execution. At each policy query, the policy combines these plan tokens with the current observation to predict robot actions. This formulation enables the policy to extract useful motion cues from human videos. We find this formulation transfers exceptionally well, even when human demonstration motions are infeasible for robots. In controlled simulation and real-world evaluations, 3PoinTr achieves an average success rate 25.0 percentage points higher than the strongest behavior-cloning and video-pretraining baselines on real-world tasks, and an improvement of 29.6 percentage points in simulation. Ablations establish the benefits of our method, including predicting non-embodiment 3D tracks once as a global plan, Perceiver-IO compression, and a residual cross-attention mechanism. Our point-track prediction network also outperforms a strong 3D-flow baseline by retaining supervision for temporarily occluded points. Project page: https://adamhung60.github.io/3PoinTr/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
