基于离线数据的赋能度测量用于内在动机探索
Measuring Empowerment from Offline Data for Intrinsically Motivated Exploration
AI总结:
本文提出首个无需环境交互、仅从离线未标注数据估计赋能度的方法,联合学习赋能度最大化策略与折扣状态占用度量,可改善在线目标条件强化学习的探索效果。
摘要:
机器人、编码等领域的智能体任务中,大量未标注的先验经验数据集已变得日益易得。这类任务的常见做法是利用先验数据训练策略,再通过在线 rollout 进一步优化。但尽管已有大量研究探讨如何结合离线与在线数据进行策略提取,关于如何利用先验数据在在线学习阶段高效探索的研究却少得多。此前的研究尝试通过定义赋能度(empowerment)来形式化这一思路,赋能度衡量智能体可实现的未来状态数量;赋能度更高的状态需要更多探索,因为存在更多可探索的未来。然而,现有的赋能度估计方法需要在环境中进行在线策略 rollout。我们提出了首个无需任何环境交互、仅从未标注先验数据估计赋能度的方法。通过联合学习赋能度最大化策略及其折扣状态占用度量,我们的估计器可在任意未知策略生成的未标注轨迹上训练。我们证明,我们的赋能度估计器能定性给出直观的赋能度度量,且预训练的估计器可用于改善在线目标条件强化学习场景中的探索效果。代码可在该 https URL 获取。
英文摘要:
Large, unlabeled datasets of prior experience have become increasingly available for agentic tasks in domains such as robotics and coding. A common recipe for these tasks is to train a policy using the prior data, and then further improve it with online rollouts. But while much work has studied how to use both the offline and online data for policy extraction, far less work has been done to understand how the prior data can be used to efficiently \textit{explore} during the online learning phase. Previous work has attempted to formalize this idea by defining the \textit{empowerment}, a measure of how many future states an agent can realize; states with higher empowerment require more exploration as there are more futures to explore. However, existing methods for estimating empowerment require online policy rollouts in the environment. We propose the first method for estimating empowerment from unlabeled prior data, without any environment interaction. By jointly learning an empowerment-maximizing policy together with its discounted state-occupancy measure, our estimator can be trained on unlabeled trajectories generated from any unknown policy. We demonstrate that our empowerment estimator qualitatively gives intuitive empowerment measures and show that the pretrained estimator can be used to improve exploration in the online goal-conditioned reinforcement learning setting. Code is available at \url{https://github.com/ishirgarg/offline-empowerment}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。