深度强化学习中基于可扩展观测模型的高效贝叶斯策略复用
Efficient Bayesian Policy Reuse with a Scalable Observation Model in Deep Reinforcement Learning
AI总结：
针对深度强化学习中的策略迁移问题，提出用即时状态转移样本和可扩展观测模型改进贝叶斯策略复用，并扩展至持续学习以避免负迁移。
摘要：
贝叶斯策略复用（BPR）是一种通用的策略迁移框架，它根据某些观测信号和一个经过训练的观测模型推断任务信念，从而从离线策略库中选择源策略。本文提出一种改进的BPR方法，以在深度强化学习（DRL）中实现更高效的策略迁移。首先，大多数BPR算法采用回合回报作为观测信号，该信号包含的信息有限，并且直到一个回合结束时才能获得。与之不同，我们采用信息丰富且即时可得的状态转移样本作为观测信号，以实现更快、更准确的任务推断。其次，BPR算法通常需要大量样本估计基于表格的观测模型的概率分布，这在学习和维护上代价高昂，甚至不可行，尤其是在使用状态转移样本作为信号时。因此，我们提出一种可扩展观测模型，该模型仅通过少量样本拟合源任务的状态转移函数，并能够泛化到目标任务中观测到的任意信号。此外，我们通过以即插即用方式扩展该可扩展观测模型，将离线模式的BPR扩展到持续学习场景，从而在面对新的未知任务时避免负迁移。实验结果表明，我们的方法能够持续促进更快、更高效的策略迁移。
英文摘要：
Bayesian policy reuse (BPR) is a general policy transfer framework for selecting a source policy from an offline library by inferring the task belief based on some observation signals and a trained observation model. In this paper, we propose an improved BPR method to achieve more efficient policy transfer in deep reinforcement learning (DRL). First, most BPR algorithms use the episodic return as the observation signal that contains limited information and cannot be obtained until the end of an episode. Instead, we employ the state transition sample, which is informative and instantaneous, as the observation signal for faster and more accurate task inference. Second, BPR algorithms usually require numerous samples to estimate the probability distribution of the tabular-based observation model, which may be expensive and even infeasible to learn and maintain, especially when using the state transition sample as the signal. Hence, we propose a scalable observation model based on fitting state transition functions of source tasks from only a small number of samples, which can generalize to any signals observed in the target task. Moreover, we extend the offline-mode BPR to the continual learning setting by expanding the scalable observation model in a plug-and-play fashion, which can avoid negative transfer when faced with new unknown tasks. Experimental results show that our method can consistently facilitate faster and more efficient policy transfer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
