基于潜在空间强化学习引导你的扩散策略
Steering Your Diffusion Policy with Latent Space Reinforcement Learning
AI总结：
本文提出DSRL方法，通过在扩散策略的潜在噪声空间运行强化学习，实现仅需黑盒访问且无需修改权重的样本高效现实世界自主策略适应。
摘要：
从人类演示中学习的机器人控制策略在许多现实世界应用中取得了令人印象深刻的结果。然而，在初始性能不令人满意的场景中（这在新颖的开放世界设定中经常发生），这种行为克隆（BC）学习的策略通常需要收集额外的人类演示来进一步改善其行为——这是一个昂贵且耗时的过程。相比之下，强化学习（RL）有望实现自主的在线策略改进，但由于通常需要大量样本，往往无法实现这一目标。在这项工作中，我们朝着通过高效的现实世界RL实现BC训练策略的快速自主适应迈出了步伐。我们特别关注扩散策略——一种最先进的BC方法——提出了基于强化学习的扩散引导（DSRL）：通过在BC策略的潜在噪声空间上运行RL来适应BC策略。我们表明，DSRL具有很高的样本效率，仅需要对BC策略的黑盒访问，并能实现有效的现实世界自主策略改进。此外，DSRL避免了与微调扩散策略相关的许多挑战，完全无需修改基础策略的权重。我们在模拟基准、现实世界机器人任务以及适应预训练的通用策略上展示了DSRL，说明了其在现实世界策略改进中的样本效率和有效性能。
英文摘要：
Robotic control policies learned from human demonstrations have achieved impressive results in many real-world applications. However, in scenarios where initial performance is not satisfactory, as is often the case in novel open-world settings, such behavioral cloning (BC)-learned policies typically require collecting additional human demonstrations to further improve their behavior -- an expensive and time-consuming process. In contrast, reinforcement learning (RL) holds the promise of enabling autonomous online policy improvement, but often falls short of achieving this due to the large number of samples it typically requires. In this work we take steps towards enabling fast autonomous adaptation of BC-trained policies via efficient real-world RL. Focusing in particular on diffusion policies -- a state-of-the-art BC methodology -- we propose diffusion steering via reinforcement learning (DSRL): adapting the BC policy by running RL over its latent-noise space. We show that DSRL is highly sample efficient, requires only black-box access to the BC policy, and enables effective real-world autonomous policy improvement. Furthermore, DSRL avoids many of the challenges associated with finetuning diffusion policies, obviating the need to modify the weights of the base policy at all. We demonstrate DSRL on simulated benchmarks, real-world robotic tasks, and for adapting pretrained generalist policies, illustrating its sample efficiency and effective performance at real-world policy improvement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
