arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.01971cs.ROcs.LG
收藏

运行时观测干预使视觉-语言-动作模型更具视觉鲁棒性

Run-time Observation Interventions Make Vision-Language-Action Models More Visually Robust

Princeton University(普林斯顿大学)

Asher J. Hancock, Allen Z. Ren, Anirudha Majumdar

AI总结:

针对VLA模型对干扰物体和背景颜色脆弱的问题,本文提出运行时干预方案BYOVLA,通过动态识别敏感区域并最小化修改任务无关区域,无需微调即可使模型在干扰下几乎保持原有性能。

摘要:

在大规模互联网数据和机器人演示上训练的视觉-语言-动作(VLA)模型有潜力充当通用机器人策略。然而,尽管经过了大规模训练,VLA 模型通常对与任务无关的视觉细节(如干扰物体或背景颜色)很脆弱。我们提出 Bring Your Own VLA(BYOVLA):一种运行时干预方案,它(1)动态识别输入图像中模型敏感的区域,并(2)使用自动化图像编辑工具对与任务无关的区域进行最小程度的修改,以降低模型的敏感性。我们的方法兼容任何现成的 VLA,无需模型微调或访问模型权重。在语言指令操作任务上的硬件实验表明,BYOVLA 使最先进的 VLA 模型在存在干扰物体和背景的情况下几乎保持其标称性能,否则任务成功率会下降高达 40%。更多信息、视频和代码见网站:https://aasherh.github.io/byovla/。

英文摘要:

Vision-language-action (VLA) models trained on large-scale internet data and robot demonstrations have the potential to serve as generalist robot policies. However, despite their large-scale training, VLAs are often brittle to task-irrelevant visual details such as distractor objects or background colors. We introduce Bring Your Own VLA (BYOVLA): a run-time intervention scheme that (1) dynamically identifies regions of the input image that the model is sensitive to, and (2) minimally alters task-irrelevant regions to reduce the model's sensitivity using automated image editing tools. Our approach is compatible with any off the shelf VLA without model fine-tuning or access to the model's weights. Hardware experiments on language-instructed manipulation tasks demonstrate that BYOVLA enables state-of-the-art VLA models to nearly retain their nominal performance in the presence of distractor objects and backgrounds, which otherwise degrade task success rates by up to 40%. Website with additional information, videos, and code: https://aasherh.github.io/byovla/ .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑