ELEMENT: 通过最大熵进行情景与终身探索
ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy
AI总结：
本文提出ELEMENT，一种多尺度内在动机框架，通过结合终身熵最大化和情景熵项，解决大规模无奖励探索中内在奖励消失和熵估计不可行的问题，在状态覆盖和预训练上超越现有基线。
摘要：
强化学习智能体依赖于奖励信号，而奖励信号的密度很少在设计师的控制之下，当此类信号缺失时，智能体必须产生自身的驱动力去探索。状态熵最大化为此提供了一个有原则的目标，但现有方法在大规模场景下会在两方面失效：一旦状态被访问过，内在奖励就会消失，从而不鼓励重新访问那些通往后续状态的关口；并且对数百万个累积观测进行熵估计在计算上变得不可行。我们通过情景与终身最大熵探索（ELEMENT）来解决这两个问题，这是一个多尺度的内在动机框架，用于无奖励探索并可迁移到下游任务。ELEMENT 将终身熵最大化与一个在更快时间尺度上作用的互补情景项相结合。对于情景项，我们推导出平均情景状态熵，这是一种内在奖励，它是奖励分解目标的可处理上界的精确最小化器；对于终身项，我们提出了一种基于kNN图的估计器，使得熵保持可处理且不遗忘。ELEMENT 在状态覆盖率和无监督预训练方面持续优于最先进的内在奖励基线。视频、代码和补充材料：此 https URL。
英文摘要：
Reinforcement learning agents depend on reward signals whose density is rarely under the designer's control, and when such signals are absent, an agent must generate its own drive to explore. State entropy maximization offers a principled objective for this, but existing methods break down at scale in two ways: the intrinsic reward vanishes once a state has been visited, discouraging revisits to the very gateways that lead onward, and estimating entropy over millions of accumulated observations becomes computationally prohibitive. We address both with Episodic and Lifelong Exploration via Maximum Entropy (ELEMENT), a multiscale intrinsically motivated framework for reward-free exploration that transfers to downstream tasks. ELEMENT couples lifelong entropy maximization with a complementary episodic term acting on a faster timescale. For the episodic term, we derive average episodic state entropy, an intrinsic reward that is the exact minimizer of a tractable upper bound on the reward-decomposition objective; for the lifelong term, we propose a $k$NN graph-based estimator that keeps entropy tractable without forgetting. ELEMENT consistently outperforms state-of-the-art intrinsic reward baselines on state coverage and unsupervised pre-training. Videos, code, and supplementary material: https://sites.google.com/view/element-rl.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
