不确定性优先经验回放
Uncertainty Prioritized Experience Replay
AI总结：
本文提出利用认知不确定性为回放缓冲区中的转移排序，以抑制噪声导致的错误优先级，并在玩具任务和 Atari 上验证其优于 QR-DQN 基准。
摘要：
优先经验回放通过选择相关转移来更新参数估计，从而提高样本效率，是当代基于价值的深度强化学习模型的关键组成部分。通常，转移会依据其时间差分误差被赋予优先级。然而，这种方法容易偏好含噪转移，即使在价值估计已经非常接近目标均值时也是如此。该现象类似于探索文献中假设的“噪声电视”问题：受探索引导的智能体会因把噪声误认为新颖性而陷入停滞。为缓解价值估计中噪声造成的破坏性影响，我们提出使用认知不确定性估计来指导回放缓冲区中转移的优先级排序。认知不确定性量化的是可通过学习降低的不确定性，因此可减少从缓冲区中采样到的、由不可预测随机过程生成的转移。我们首先在两个表格型玩具模型中展示认知不确定性优先回放的收益：一个简单的多臂老虎机任务和一个含噪网格世界。随后，我们在 Atari 套件上评估该优先级方案，其表现优于分位数回归深度 Q 学习基准；由此为在强化学习智能体中使用不确定性优先回放开辟了道路。
英文摘要：
Prioritized experience replay, which improves sample efficiency by selecting relevant transitions to update parameter estimates, is a crucial component of contemporary value-based deep reinforcement learning models. Typically, transitions are prioritized based on their temporal difference error. However, this approach is prone to favoring noisy transitions, even when the value estimation closely approximates the target mean. This phenomenon resembles the noisy TV problem postulated in the exploration literature, in which exploration-guided agents get stuck by mistaking noise for novelty. To mitigate the disruptive effects of noise in value estimation, we propose using epistemic uncertainty estimation to guide the prioritization of transitions from the replay buffer. Epistemic uncertainty quantifies the uncertainty that can be reduced by learning, hence reducing transitions sampled from the buffer generated by unpredictable random processes. We first illustrate the benefits of epistemic uncertainty prioritized replay in two tabular toy models: a simple multi-arm bandit task, and a noisy gridworld. Subsequently, we evaluate our prioritization scheme on the Atari suite, outperforming quantile regression deep Q-learning benchmarks; thus forging a path for the use of uncertainty prioritized replay in reinforcement learning agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
