arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.15917cs.RO
收藏

犯错是机器人的特性:部署期间基于价值的快速试错

To Err is Robotic: Rapid Value-Based Trial-and-Error during Deployment

Stanford University(斯坦福大学)

Maximilian Du, Alexander Khazatsky, Tobias Gerstenberg, Chelsea Finn

AI总结:

针对机器人在新场景下首次尝试易失败、无方向重试易出不可逆错误的问题,提出Bellman-Guided Retrials算法,通过价值函数监控进度、调整策略,在仿真和真实场景分别将基础策略成功率提升20%、50%以上。

摘要:

当遇到全新场景时,机器人很难在首次尝试时就取得成功。在这些具有挑战性的情境中,知晓如何快速且有意义地重试至关重要。在多样化数据上训练的机器人可以自然产生重试行为,但这类机器人策略通常只会表现出无方向的重试行为,并且可能在发生不可挽回的错误之前不会终止次优的执行方案。我们可以通过为机器人策略注入明确的尝试、评估并重试多种不同策略的能力来对其进行改进。\n我们提出了Bellman-Guided Retrials(贝尔曼引导重试算法),该算法在基础机器人策略之上运行,通过监控机器人的进度,检测何时需要改变计划,并调整执行的策略,直到机器人成功完成任务。我们首先以在多种场景下的专家演示数据训练得到的基础策略为起点,随后使用相同的专家演示数据训练一个价值函数,用于估计任务的完成进度。在测试阶段,我们利用该价值函数将预期进度速率与实际达到的进度速率进行对比。如果当前策略无法以合理的速率取得进展,我们会让机器人恢复到安全状态,并从基础策略中采样新的策略,同时让新策略偏向于避开近期失败过的行为。\n我们在包含多种场景的仿真环境和真实世界环境中对所提方法进行了评估。实验结果表明,Bellman-Guided Retrials将基础策略的平均绝对成功率在仿真环境中提升了20%以上,在真实世界实验中提升了50%以上,证明了这一为现有已训练策略注入明确试错能力的框架具有良好的应用前景。如需查看评估视频和其他文档,请访问https://sites.google.com/view/to-err-robotic/home

英文摘要:

When faced with a novel scenario, it can be hard to succeed on the first attempt. In these challenging situations, it is important to know how to retry quickly and meaningfully. Retrying behavior can emerge naturally in robots trained on diverse data, but such robot policies will typically only exhibit undirected retrying behavior and may not terminate a suboptimal approach before an unrecoverable mistake. We can improve these robot policies by instilling an explicit ability to try, evaluate, and retry a diverse range of strategies. We introduce Bellman-Guided Retrials, an algorithm that works on top of a base robot policy by monitoring the robot's progress, detecting when a change of plan is needed, and adapting the executed strategy until the robot succeeds. We start with a base policy trained on expert demonstrations of a variety of scenarios. Then, using the same expert demonstrations, we train a value function to estimate task completion. During test time, we use the value function to compare our expected rate of progress to our achieved rate of progress. If our current strategy fails to make progress at a reasonable rate, we recover the robot and sample a new strategy from the base policy while skewing it away from behaviors that have recently failed. We evaluate our method on simulated and real-world environments that contain a diverse suite of scenarios. We find that Bellman-Guided Retrials increases the average absolute success rates of base policies by more than 20% in simulation and 50% in real-world experiments, demonstrating a promising framework for instilling existing trained policies with explicit trial and error capabilities. For evaluation videos and other documentation, go to https://sites.google.com/view/to-err-robotic/home

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑