arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.16075cs.LGcs.AIcs.RO
收藏

不要从零开始:通过基于插值的策略扩散实现行为优化

Don't Start from Scratch: Behavioral Refinement via Interpolant-based Policy Diffusion

National University of Singapore(新加坡国立大学);Smart Systems Institute, NUS(新加坡国立大学智能系统研究所)

Kaiqi Chen, Eugene Lim, Kelvin Lin, Yiyang Chen, Harold Soh

AI总结:

针对扩散模型用于模仿学习时因源分布为高斯噪声导致少扩散步、小数据下性能差的问题,提出基于随机插值框架的BRIDGER方法,可桥接任意策略,在仿真和真实机器人实验中性能优于现有最优扩散策略。

摘要:

模仿学习使智能体能够通过从演示中学习来模仿行为。近年来,具备高维多模态分布建模能力的扩散模型在模仿学习任务中展现出出色性能,这类模型通过将动作(或状态)从标准高斯噪声扩散来学习塑造策略。然而,待学习的目标策略通常与高斯分布差异显著,这种不匹配会导致在使用少量扩散步(以提升推理速度)以及数据有限的情况下性能不佳。本研究的核心思路是,从比高斯分布更具信息量的源分布出发,能够让扩散方法缓解上述局限。我们提出了理论结果、新方法以及实证发现,证明了使用有信息量的源策略的优势。我们提出的名为BRIDGER的方法,利用随机插值(stochastic interpolants)框架桥接任意策略,从而实现灵活的模仿学习方案。它对先前工作进行了泛化:标准高斯分布仍可使用,但若有其他源策略也可加以利用。在具有挑战性的仿真基准以及真实机器人实验中,BRIDGER的表现优于当前最优的扩散策略。我们还进一步分析了应用BRIDGER时的设计考量因素。BRIDGER的代码可在https://github.com/clear-nus/bridger获取。

英文摘要:

Imitation learning empowers artificial agents to mimic behavior by learning from demonstrations. Recently, diffusion models, which have the ability to model high-dimensional and multimodal distributions, have shown impressive performance on imitation learning tasks. These models learn to shape a policy by diffusing actions (or states) from standard Gaussian noise. However, the target policy to be learned is often significantly different from Gaussian and this mismatch can result in poor performance when using a small number of diffusion steps (to improve inference speed) and under limited data. The key idea in this work is that initiating from a more informative source than Gaussian enables diffusion methods to mitigate the above limitations. We contribute both theoretical results, a new method, and empirical findings that show the benefits of using an informative source policy. Our method, which we call BRIDGER, leverages the stochastic interpolants framework to bridge arbitrary policies, thus enabling a flexible approach towards imitation learning. It generalizes prior work in that standard Gaussians can still be applied, but other source policies can be used if available. In experiments on challenging simulation benchmarks and on real robots, BRIDGER outperforms state-of-the-art diffusion policies. We provide further analysis on design considerations when applying BRIDGER. Code for BRIDGER is available at https://github.com/clear-nus/bridger.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑