零样本长时程灵巧操作：基于多视图3D接地VLM推理
ZeroDex: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning
AI总结：
提出零样本框架，利用多视图RGB图像通过VLM生成3D任务规划，结合三角测量和射线投票实现精确3D接地，支持抓取和工具使用，在真实实验中优于基线方法。
摘要：
我们提出了一个零样本框架，用于长时程灵巧操作，该框架将语言指令从校准的多视图RGB图像接地到可执行的3D任务规划。我们的系统不是训练端到端策略，而是使用视觉语言模型（VLM）生成参考帧任务接地和原始级2D关键点，然后通过多视图融合将其提升到3D。这种提升结合了视图级VLM接地的三角测量与参考视图射线投票，后者沿语义相机射线搜索跨相邻视图的几何一致候选点。生成的3D关键点支持抓取和放置以及工具使用：对于工具使用，我们检索与推断技能类别对应的以对象为中心的原子动作，并将其存储的6D工具轨迹对齐到场景；对于灵巧执行，我们将提升的抓取关键点扩展为任务条件抓取可行区域，并使用臂手运动生成器生成可行的抓取-运动对。真实世界实验表明，与单视图RGB-D接地和微调VLA基线相比，3D接地精度和执行可靠性有所提高。我们进一步通过闭环状态验证和重新规划展示了长时程操作，实现了在新场景中对未见物体和工具使用任务的零样本执行。
英文摘要：
We present ZeroDex, a zero-shot framework for long-horizon dexterous manipulation that grounds language instructions into executable 3D task plans from calibrated multi-view RGB images. Rather than training an end-to-end policy, our system uses a vision-language model (VLM) to produce reference-frame task grounding and primitive-level 2D keypoints, then lifts them into 3D via multi-view fusion. This lifting combines triangulation of view-wise VLM groundings with reference-view ray voting, which searches along a semantic camera ray for geometrically consistent candidates across neighboring views. The resulting 3D keypoints support both pick-and-place and tool-use: for tool-use, we retrieve an object-centric atomic action corresponding to the inferred skill category and align its stored 6D tool trajectory to the scene; for dexterous execution, we expand the lifted grasp keypoint into a task-conditioned grasp affordance region and generate feasible grasp-motion pairs with an arm-hand motion generator. Real-world experiments show improved 3D grounding accuracy and execution reliability over single-view RGB-D grounding and fine-tuned VLA baselines. We further demonstrate long-horizon manipulation through closed-loop status verification and replan, enabling zero-shot execution on unseen objects and tool-use tasks in novel scenes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
