TacZero：使用带触觉反馈的通用视觉语言模型实现免训练插销任务
TacZero: Training-Free Peg Insertion Using a General-Purpose Vision-Language Model with Tactile Feedback
浏览论文内容
中文总结 AI 辅助
TacZero利用预训练通用视觉语言模型，结合视觉与触觉观测，无需额外训练或任务规则即可选择动作，实现免训练的插销任务，实验证明触觉输入显著提升成功率。
中文摘要 AI 辅助
能够根据语言指令和感官观测自主决定动作的机器人，可以在无需针对特定任务进行训练或手工设计规则的情况下，执行新的接触密集型操作任务。为了执行这些任务，机器人必须推断物体之间如何接触以及由此产生的运动，然后选择动作。在接触推断和动作选择方面，先前的方法涉及设计估计模型和触觉反馈控制律，或从触觉数据中学习用于物体运动估计、动作结果预测和动作选择的模型。相反，我们提出了TacZero，它使用预训练的通用视觉语言模型（VLM）来解释视觉和触觉观测，并选择机器人动作，而无需额外的触觉或操作训练，也无需针对接触解释或动作选择的特定任务规则。TacZero向VLM提供相机图像、机器人状态以及以数值或叠加在图像上的向量形式表示的三轴触觉响应。根据这些观测和交互历史，VLM生成指定目标末端执行器位置和夹爪开闭的命令，由低级控制器执行。在真实世界的圆柱插销实验中，TacZero在数值触觉输入下20次试验成功15次，而无触觉输入时20次成功10次。本研究为使用通用VLM进行接触密集型操作的进一步研究提供了具体起点，并强调了在这一方向上推进所面临的挑战。
英文摘要
Robots that autonomously determine their actions from language instructions and sensory observations could perform new contact-rich manipulation tasks without task-specific training or hand-designed rules. To perform these tasks, robots must infer how objects contact one another and move as a result, then select actions. For contact inference and action selection, prior approaches involve designing estimation models and tactile feedback control laws, or learning models for object-motion estimation, action-outcome prediction, and action selection from tactile data. Instead, we propose TacZero, which uses a pretrained general-purpose vision-language model (VLM) to interpret visual and tactile observations and select robot actions without additional tactile or manipulation training or task-specific rules for contact interpretation or action selection. TacZero provides the VLM with camera images, robot state, and three-axis tactile responses represented as numerical values or vectors overlaid on the images. From these observations and interaction history, the VLM generates commands specifying target end-effector positions and gripper opening or closing, which a low-level controller executes. In real-world cylindrical-peg insertion experiments, TacZero succeeded in 15 of 20 trials with numerical tactile input, compared with 10 of 20 without tactile input. This study provides a concrete starting point for further research on contact-rich manipulation using general-purpose VLMs and highlights challenges in pursuing this direction.
发表机构
- OMRON SINIC X Corporation（欧姆龙SINIC X公司）
机构由 AI 辅助整理，请以论文原文为准。
