arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.09990cs.LGcs.NEcs.RO
收藏

SHIRE:在强化学习中利用人类直觉提升样本效率

SHIRE: Enhancing Sample Efficiency using Human Intuition in REinforcement Learning

Purdue University(普渡大学)

Amogh Joshi, Adarsh Kumar Kosta, Kaushik Roy

AI总结:

针对深度强化学习样本效率低、可解释性不足的问题,提出融合概率图模型编码人类直觉的SHIRE框架,在多环境实现25%-78%的样本效率提升,同时增强策略可解释性并经真实场景验证有效。

摘要:

近年来,神经网络在深度估计、光流估计、同步定位与地图构建(SLAM)以及自动控制等机器人感知与控制任务中展现出的能力,推动了其广泛应用。深度强化学习因无需承担监督学习那类难以承受的训练成本,已在这些场景中得到大量使用。但深度强化学习存在样本效率低下的问题,即需要大量与环境的交互才能收敛到可接受的解。Deep Q Learning、Soft Actor-Critic等现代强化学习算法试图弥补这一缺陷,却无法提供自主机器人等应用所需的可解释性。人类能凭直觉理解机器人领域中常见的长时程序列任务,合理利用这类直觉可让强化学习策略更具可解释性,同时提升样本效率。本研究提出SHIRE——一种利用概率图模型(PGMs)编码人类直觉、并将其融入深度强化学习训练流程以提升样本效率的新型框架。在我们评估的所有环境中,该框架以可忽略的额外开销实现了25%-78%的样本效率提升。此外,通过向强化学习智能体传授编码后的基础行为,SHIRE提升了策略的可解释性。一项真实场景演示进一步验证了采用该框架训练的策略的有效性。

英文摘要:

The ability of neural networks to perform robotic perception and control tasks such as depth and optical flow estimation, simultaneous localization and mapping (SLAM), and automatic control has led to their widespread adoption in recent years. Deep Reinforcement Learning has been used extensively in these settings, as it does not have the unsustainable training costs associated with supervised learning. However, DeepRL suffers from poor sample efficiency, i.e., it requires a large number of environmental interactions to converge to an acceptable solution. Modern RL algorithms such as Deep Q Learning and Soft Actor-Critic attempt to remedy this shortcoming but can not provide the explainability required in applications such as autonomous robotics. Humans intuitively understand the long-time-horizon sequential tasks common in robotics. Properly using such intuition can make RL policies more explainable while enhancing their sample efficiency. In this work, we propose SHIRE, a novel framework for encoding human intuition using Probabilistic Graphical Models (PGMs) and using it in the Deep RL training pipeline to enhance sample efficiency. Our framework achieves 25-78% sample efficiency gains across the environments we evaluate at negligible overhead cost. Additionally, by teaching RL agents the encoded elementary behavior, SHIRE enhances policy explainability. A real-world demonstration further highlights the efficacy of policies trained using our framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑