优先化生成回放
Prioritized Generative Replay
AI总结：
本文提出优先化生成回放方法，用条件扩散模型对在线经验进行参数化记忆和稠密化，并通过相关性函数引导生成更有用的转移，从而提升在线强化学习的性能和样本效率。
摘要：
样本高效的在线强化学习通常使用回放缓冲区来存储经验，以便在更新价值函数时重用。然而，均匀回放是低效的，因为某些类别的转移可能与学习更相关。虽然对更有用样本进行优先化是有帮助的，但这种策略也可能导致过拟合，因为有用样本往往更加稀有。在这项工作中，我们提出了一种优先化的、参数化的智能体记忆版本，使用生成模型来捕获在线经验。这一范式能够实现：（1）对过去经验进行稠密化，新生成的样本受益于生成模型的泛化能力；（2）通过一族“相关性函数”进行引导，将这些生成样本推向智能体已获取历史中更有用的部分。我们展示了这一方法可以使用条件扩散模型以及简单相关性函数（如基于好奇心或基于价值的度量）来实例化。我们的方法在基于状态和基于像素的领域中都一致地提高了性能和样本效率。我们揭示了这些收益背后的机制，展示了引导如何促进生成转移中的多样性并减少过拟合。我们还展示了我们的方法能够以比之前更高的更新数据比来训练策略，为更好地扩展在线强化学习智能体开辟了途径。
英文摘要：
Sample-efficient online reinforcement learning often uses replay buffers to store experience for reuse when updating the value function. However, uniform replay is inefficient, since certain classes of transitions can be more relevant to learning. While prioritization of more useful samples is helpful, this strategy can also lead to overfitting, as useful samples are likely to be more rare. In this work, we instead propose a prioritized, parametric version of an agent's memory, using generative models to capture online experience. This paradigm enables (1) densification of past experience, with new generations that benefit from the generative model's generalization capacity and (2) guidance via a family of "relevance functions" that push these generations towards more useful parts of an agent's acquired history. We show this recipe can be instantiated using conditional diffusion models and simple relevance functions such as curiosity- or value-based metrics. Our approach consistently improves performance and sample efficiency in both state- and pixel-based domains. We expose the mechanisms underlying these gains, showing how guidance promotes diversity in our generated transitions and reduces overfitting. We also showcase how our approach can train policies with even higher update-to-data ratios than before, opening up avenues to better scale online RL agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
