SLOPE：基于模型的强化学习中的乐观势能塑造

Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shaping

In this work, we study the simple yet universally applicable case of reward shaping in value-based Deep Reinforcement Learning (DRL). We show that reward shifting in the form of the linear transformation is equivalent to changing the initialization of the $Q$-function in function approximation. Based on such an equivalence, we bring the key insight that a positive reward shifting leads to conservative exploitation, while a negative reward shifting leads to curiosity-driven exploration. Accordingly, conservative exploitation improves offline RL value estimation, and optimistic value estimation …

乐观策略正则化

Optimistic Policy Regularization

OPR通过保留和强化历史成功轨迹提升深度强化学习的样本效率和最终性能

Mastering 2048 with Delayed Temporal Coherence Learning, Multi-Stage Weight Promotion, Redundant Encoding and Carousel Shaping

2048 is an engaging single-player, nondeterministic video puzzle game, which, thanks to the simple rules and hard-to-master gameplay, has gained massive popularity in recent years. As 2048 can be conveniently embedded into the discrete-state Markov decision processes framework, we treat it as a testbed for evaluating existing and new methods in reinforcement learning. With the aim to develop a strong 2048 playing program, we employ temporal difference learning with systematic n-tuple networks. We show that this basic method can be significantly improved with temporal coherence learning, multi-…

面向目标条件强化学习的基于磁场的奖励塑形

Magnetic Field-Based Reward Shaping for Goal-Conditioned Reinforcement Learning

针对动态目标与障碍物下目标条件强化学习的稀疏奖励问题，提出基于磁场非线性各向异性强度的MFRS奖励塑形，并通过辅助势函数保持最优策略不变性，显著提升样本效率。

Exploiting Uncertainty of Loss Landscape for Stochastic Optimization

We introduce novel variants of momentum by incorporating the variance of the stochastic loss function. The variance characterizes the confidence or uncertainty of the local features of the averaged loss surface across the i.i.d. subsets of the training data defined by the mini-batches. We show two applications of the gradient of the variance of the loss function. First, as a bias to the conventional momentum update to encourage conformity of the local features of the loss function (e.g. local minima) across mini-batches to improve generalization and the cumulative training progress made per ep…

ISEP: 通过随机策略优化实现离线强化学习的隐式支持扩展

ISEP: Implicit Support Expansion for Offline Reinforcement Learning via Stochastic Policy Optimization

本文提出ISEP方法，通过随机策略优化实现离线强化学习中的隐式支持扩展，以解决传统方法在安全约束下难以发现最优行为的问题，核心贡献是通过价值函数插值和随机动作选择策略提高策略改进的导航能力。