arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.07290cs.LGcs.AI
收藏

面向控制的基于条件扩散模型的价值函数估计

Value function estimation using conditional diffusion models for control

Apple(苹果公司)

Bogdan Mazoure, Walter Talbott, Miguel Angel Bautista, Devon Hjelm, Alexander Toshev, Josh Susskind

AI总结:

针对深度强化学习高质量演示数据不足的问题,提出基于扩散模型的DVF算法,可利用无奖励无动作的低质量状态序列学习交互动力学并估计动作价值,在机器人基准上取得良好效果。

摘要:

深度强化学习中一个相当可靠的趋势是,只要训练数据量同步扩展,性能会随参数数量提升。随着对大模型的需求增长,我们必须尽早解决高质量演示数据可能耗尽的潜在问题。在这种情况下,与其仅通过成本高昂的人类演示收集新数据,或冒着效果不确定的风险进行仿真到真实的迁移,利用大量现成的低质量数据会更有益。由于行为克隆、时序差分学习等经典控制算法无法直接用于无奖励或无动作的数据,该解决方案需要适用于连续控制的新型训练范式。我们提出了一种名为Diffused Value Function(DVF,扩散价值函数)的简单算法,它使用扩散模型学习环境-机器人交互动力学的联合多步模型。该模型可从状态序列(即无需访问奖励函数和动作)中高效学习,随后可直接用于估计每个动作的价值。我们展示了DVF如何被用于高效捕获多个控制器的状态访问度量,并在具有挑战性的机器人基准测试中取得了可观的定性与定量结果。

英文摘要:

A fairly reliable trend in deep reinforcement learning is that the performance scales with the number of parameters, provided a complimentary scaling in amount of training data. As the appetite for large models increases, it is imperative to address, sooner than later, the potential problem of running out of high-quality demonstrations. In this case, instead of collecting only new data via costly human demonstrations or risking a simulation-to-real transfer with uncertain effects, it would be beneficial to leverage vast amounts of readily-available low-quality data. Since classical control algorithms such as behavior cloning or temporal difference learning cannot be used on reward-free or action-free data out-of-the-box, this solution warrants novel training paradigms for continuous control. We propose a simple algorithm called Diffused Value Function (DVF), which learns a joint multi-step model of the environment-robot interaction dynamics using a diffusion model. This model can be efficiently learned from state sequences (i.e., without access to reward functions nor actions), and subsequently used to estimate the value of each action out-of-the-box. We show how DVF can be used to efficiently capture the state visitation measure for multiple controllers, and show promising qualitative and quantitative results on challenging robotics benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑