arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.01693cs.LG
收藏

无模型和基于模型的强化学习的变分推理

Variational Inference for Model-Free and Model-Based Reinforcement Learning

Felix Leibfried

摘要:

变分推理(VI)是一种特殊类型的近似贝叶斯推理,它用一个易处理的分布来近似一个难处理的后验分布。VI将推理问题转换为优化问题,更具体地说,目标是最大化关于近似后验的参数的边际似然的对数的下限。另一方面,强化学习(RL)处理的是自主智能体,以及如何使它们的行为最优,例如最大化某些预期未来累积奖励的概念。在非顺序设置中,代理人的行为不会对环境的未来状态产生影响,RL被上下文强盗和贝叶斯优化所覆盖。然而,在一个适当的顺序场景中,代理人的行为会影响未来的状态,需要仔细权衡瞬时奖励与潜在的长期奖励。这份手稿显示了VI和RL的明显不同的主题是如何以两种基本方式联系在一起的。首先,RL的优化目标,以最大限度地提高未来的累积回报,可以恢复通过VI目标下的软政策约束,在非顺序和顺序设置。这种策略约束不仅仅是人为的,而且已被证明是许多RL任务中有用的正则化器,可以显着提高代理性能。其次,在基于模型的强化学习中,代理的目标是了解他们所处的环境,模型学习部分可以自然地被描述为控制环境动态过程的推理问题。对于后者,我们将区分两种情况:第六种情况是环境状态完全可被智能体观察到,第六种情况是环境状态只能通过观察分布部分观察到。

英文摘要:

Variational inference (VI) is a specific type of approximate Bayesian inference that approximates an intractable posterior distribution with a tractable one. VI casts the inference problem as an optimization problem, more specifically, the goal is to maximize a lower bound of the logarithm of the marginal likelihood with respect to the parameters of the approximate posterior. Reinforcement learning (RL) on the other hand deals with autonomous agents and how to make them act optimally such as to maximize some notion of expected future cumulative reward. In the non-sequential setting where agents' actions do not have an impact on future states of the environment, RL is covered by contextual bandits and Bayesian optimization. In a proper sequential scenario, however, where agents' actions affect future states, instantaneous rewards need to be carefully traded off against potential long-term rewards. This manuscript shows how the apparently different subjects of VI and RL are linked in two fundamental ways. First, the optimization objective of RL to maximize future cumulative rewards can be recovered via a VI objective under a soft policy constraint in both the non-sequential and the sequential setting. This policy constraint is not just merely artificial but has proven as a useful regularizer in many RL tasks yielding significant improvements in agent performance. And second, in model-based RL where agents aim to learn about the environment they are operating in, the model-learning part can be naturally phrased as an inference problem over the process that governs environment dynamics. We are going to distinguish between two scenarios for the latter: VI when environment states are fully observable by the agent and VI when they are only partially observable through an observation distribution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑