arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.07875cs.ROcs.AIcs.CLcs.CVcs.LG
收藏

生成式图像作为动作模型

Generative Image as Action Models

Dyson Robot Learning Lab(戴森机器人学习实验室)

Mohit Shridhar, Yat Long Lo, Stephen James

AI总结:

该研究提出GENIMA行为克隆智能体,通过微调Stable Diffusion将关节动作绘制为RGB图像目标再映射为关节序列,在25项RLBench和9项真实操作任务中,其生成的策略优于现有视觉运动方法,鲁棒性与泛化性更强,性能可比3D智能体。

摘要:

图像生成扩散模型已通过微调解锁了图像编辑、新视角合成等新能力。我们能否类似地解锁图像生成模型用于视觉运动控制?我们提出GENIMA——一种行为克隆智能体,它对Stable Diffusion(稳定扩散模型)进行微调,以在RGB图像上“绘制关节动作”作为目标。这些图像被输入控制器,由控制器将视觉目标映射为关节位置序列。我们在25项RLBench任务和9项真实世界操作任务上研究GENIMA。研究发现,通过将动作提升到图像空间,互联网预训练扩散模型能生成优于当前最优视觉运动方法的策略,尤其在对场景扰动的鲁棒性和对新物体的泛化能力上表现突出。我们的方法即便缺乏深度、关键点或运动规划器等先验信息,性能也能与3D智能体相媲美。

英文摘要:

Image-generation diffusion models have been fine-tuned to unlock new capabilities such as image-editing and novel view synthesis. Can we similarly unlock image-generation models for visuomotor control? We present GENIMA, a behavior-cloning agent that fine-tunes Stable Diffusion to 'draw joint-actions' as targets on RGB images. These images are fed into a controller that maps the visual targets into a sequence of joint-positions. We study GENIMA on 25 RLBench and 9 real-world manipulation tasks. We find that, by lifting actions into image-space, internet pre-trained diffusion models can generate policies that outperform state-of-the-art visuomotor approaches, especially in robustness to scene perturbations and generalizing to novel objects. Our method is also competitive with 3D agents, despite lacking priors such as depth, keypoints, or motion-planners.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑