用于离线强化学习的置信度条件价值函数
Confidence-Conditioned Value Functions for Offline Reinforcement Learning
AI总结：
本文提出置信度条件价值函数，使离线强化学习策略能在评估时动态调整保守程度，并在多个离散控制任务上优于现有保守离线RL算法。
摘要：
离线强化学习（RL）有望仅利用现有的静态数据集学习有效策略，而无需任何昂贵的在线交互。为此，离线RL方法必须处理数据集与所学策略之间的分布偏移。最常见的方法是学习保守的或下界的价值函数，这些函数会低估分布外（OOD）动作的回报。然而，这类方法存在一个显著缺点：基于此类价值函数优化的策略只能按照固定的、可能次优的保守程度行事。不过，如果我们能够在训练时学习适应不同保守程度的策略，并设计一种在评估时动态选择其中之一的方法，这一问题就可以得到缓解。为此，在本工作中，我们提出学习额外以保守程度为条件的价值函数，我们称之为置信度条件价值函数。我们推导出一种新形式的贝尔曼备份，它能够以高概率同时学习任意置信度下的Q值。通过以置信度为条件，我们的价值函数能够在在线评估期间利用迄今为止的观测历史来控制置信水平，从而实现自适应策略。该方法在实践中可以通过将现有保守算法的Q函数以置信度为条件来实现。我们从理论上证明，所学习的价值函数能够在任意期望置信度下产生对真实价值的保守估计。最后，我们通过实验表明，我们的算法在多个离散控制领域上优于现有的保守离线RL算法。
英文摘要：
Offline reinforcement learning (RL) promises the ability to learn effective policies solely using existing, static datasets, without any costly online interaction. To do so, offline RL methods must handle distributional shift between the dataset and the learned policy. The most common approach is to learn conservative, or lower-bound, value functions, which underestimate the return of out-of-distribution (OOD) actions. However, such methods exhibit one notable drawback: policies optimized on such value functions can only behave according to a fixed, possibly suboptimal, degree of conservatism. However, this can be alleviated if we instead are able to learn policies for varying degrees of conservatism at training time and devise a method to dynamically choose one of them during evaluation. To do so, in this work, we propose learning value functions that additionally condition on the degree of conservatism, which we dub confidence-conditioned value functions. We derive a new form of a Bellman backup that simultaneously learns Q-values for any degree of confidence with high probability. By conditioning on confidence, our value functions enable adaptive strategies during online evaluation by controlling for confidence level using the history of observations thus far. This approach can be implemented in practice by conditioning the Q-function from existing conservative algorithms on the confidence.We theoretically show that our learned value functions produce conservative estimates of the true value at any desired confidence. Finally, we empirically show that our algorithm outperforms existing conservative offline RL algorithms on multiple discrete control domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
