基于物理推理与坐标系标注的无约束强力技能获取
Unfettered Forceful Skill Acquisition with Physical Reasoning and Coordinate Frame Labeling
AI总结：
本文提出通过坐标系视觉标注引导 VLM 输出力旋量并进行物理推理，使其无需预训练即可零样本完成多种机器人强力操作，还能利用交互反馈恢复失败。
摘要：
视觉语言模型（VLM）展现出关于物理世界的丰富知识，包括对物理和空间属性、可供性以及运动的直觉。通过微调，VLM 还可以原生生成机器人轨迹。我们证明，引导 VLM 输出力旋量（wrenches）而非轨迹，可使其显式推理力，并在无需预训练的情况下，在一系列操作任务中实现零样本泛化。我们通过在机器人搭载相机图像上叠加相关坐标系的一致视觉表示来增强查询，从而实现这一点。首先，我们展示这一添加如何促成一个多功能运动控制框架，并在四项任务（打开和关闭盖子、推杯子或椅子）中评估，涵盖平移和旋转运动、一个数量级的力和位置大小、不同相机视角、标注方案以及两个机器人平台，共进行 220 多次实验，四项任务总体成功率为 51%。随后，我们证明所提出框架使 VLM 能够持续推理交互反馈，以在有人或无人监督的情况下从任务失败或未完成状态中恢复。最后，我们观察到，结合视觉标注和具身推理的提示方案可以绕过 VLM 安全防护。我们刻画了各提示组件对诱发有害行为的贡献，并讨论其对发展具身推理的影响。我们的代码、视频和数据可在 https://scalingforce.github.io/ 获取。
英文摘要：
Vision language models (VLMs) exhibit vast knowledge of the physical world, including intuition of physical and spatial properties, affordances, and motion. With fine-tuning, VLMs can also natively produce robot trajectories. We demonstrate that eliciting wrenches, not trajectories, allows VLMs to explicitly reason about forces and leads to zero-shot generalization in a series of manipulation tasks without pretraining. We achieve this by overlaying a consistent visual representation of relevant coordinate frames on robot-attached camera images to augment our query. First, we show how this addition enables a versatile motion control framework evaluated across four tasks (opening and closing a lid, pushing a cup or chair) spanning prismatic and rotational motion, an order of force and position magnitude, different camera perspectives, annotation schemes, and two robot platforms over 220 experiments, resulting in 51% success across the four tasks. Then, we demonstrate that the proposed framework enables VLMs to continually reason about interaction feedback to recover from task failure or incompletion, with and without human supervision. Finally, we observe that prompting schemes with visual annotation and embodied reasoning can bypass VLM safeguards. We characterize prompt component contribution to harmful behavior elicitation and discuss its implications for developing embodied reasoning. Our code, videos, and data are available at: https://scalingforce.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
