可泛化的粗到细机器人操作 via 语言对齐的3D关键点
Generalizable Coarse-to-Fine Robot Manipulation via Language-Aligned 3D Keypoints
AI总结：
CLAP通过任务分解、VLM微调和3D感知表示，提升机器人操作在新指令和环境下的泛化能力，实现更高的样本效率和操作精度。
摘要：
层次化的粗到细策略，其中粗分支预测感兴趣区域以引导细粒度动作预测器，已在机器人3D操作任务中展现出显著潜力，特别是在增强样本效率和实现更精确操作方面。然而，即使增强了预训练模型，这些层次策略仍然面临泛化问题。为增强对新指令和环境变化的泛化能力，我们提出了粗到细语言对齐操作策略（CLAP），该框架集成了三个关键组件：1）任务分解，2）为3D关键点预测进行VLM微调，3）3D感知表示。通过在仿真和真实机器人上的全面实验，我们展示了其卓越的泛化能力。具体而言，在GemBench上，一个针对泛化评估设计的基准上，我们的方法在使用仅1/5的训练轨迹时，平均成功率达到SOTA方法的12%更高。在现实世界实验中，我们的策略仅在10个演示上训练，就能泛化到新的指令和环境。
英文摘要：
Hierarchical coarse-to-fine policy, where a coarse branch predicts a region of interest to guide a fine-grained action predictor, has demonstrated significant potential in robotic 3D manipulation tasks by especially enhancing sample efficiency and enabling more precise manipulation. However, even augmented with pre-trained models, these hierarchical policies still suffer from generalization issues. To enhance generalization to novel instructions and environment variations, we propose Coarse-to-fine Language-Aligned manipulation Policy (CLAP), a framework that integrates three key components: 1) task decomposition, 2) VLM fine-tuning for 3D keypoint prediction, and 3) 3D-aware representation. Through comprehensive experiments in simulation and on a real robot, we demonstrate its superior generalization capability. Specifically, on GemBench, a benchmark designed for evaluating generalization, our approach achieves a 12\% higher average success rate than the SOTA method while using only 1/5 of the training trajectories. In real-world experiments, our policy, trained on only 10 demonstrations, successfully generalizes to novel instructions and environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
