arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2309.15293cs.LGcond-mat.stat-mechcs.AIcs.RO
收藏

最大扩散强化学习

Maximum diffusion reinforcement learning

Northwestern University(西北大学)

Thomas A. Berrueta, Allison Pinosky, Todd D. Murphey

AI总结:

本文提出最大扩散强化学习,利用遍历过程的统计力学去相关智能体体验,实现单次学习,推广最大熵方法,并在基准测试中超越现有性能。

摘要:

机器人和动物都通过它们的身体和感官来体验世界。它们的具身性限制了它们的体验,确保体验在空间和时间上连续展开。因此,具身智能体的体验本质上是相关的。相关性给机器学习带来了根本性的挑战,因为大多数技术依赖于数据独立同分布的假设。在强化学习中,数据直接从智能体的顺序体验中收集,违反这一假设往往是不可避免的。在这里,我们推导出一种方法,通过利用遍历过程的统计力学来克服这个问题,我们称之为最大扩散强化学习。通过去相关智能体的体验,我们的方法在单次任务尝试过程中,可证明地实现了连续部署中的单次学习。此外,我们证明我们的方法推广了著名的最大熵技术,并在流行的基准测试中稳健地超越了最先进的性能。我们在物理学、学习和控制的交汇处取得的成果,为具身强化学习智能体中透明和可靠的决策奠定了基础。

英文摘要:

Robots and animals both experience the world through their bodies and senses. Their embodiment constrains their experiences, ensuring they unfold continuously in space and time. As a result, the experiences of embodied agents are intrinsically correlated. Correlations create fundamental challenges for machine learning, as most techniques rely on the assumption that data are independent and identically distributed. In reinforcement learning, where data are directly collected from an agent's sequential experiences, violations of this assumption are often unavoidable. Here, we derive a method that overcomes this issue by exploiting the statistical mechanics of ergodic processes, which we term maximum diffusion reinforcement learning. By decorrelating agent experiences, our approach provably enables single-shot learning in continuous deployments over the course of individual task attempts. Moreover, we prove our approach generalizes well-known maximum entropy techniques, and robustly exceeds state-of-the-art performance across popular benchmarks. Our results at the nexus of physics, learning, and control form a foundation for transparent and reliable decision-making in embodied reinforcement learning agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑