arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.07178cs.LGcs.AI
收藏

超越期望回报:评估强化学习算法时考虑策略可复现性

Beyond Expected Return: Accounting for Policy Reproducibility when Evaluating Reinforcement Learning Algorithms

Imperial College London(帝国理工学院)

Manon Flageat, Bryan Lim, Antoine Cully

AI总结:

本文针对强化学习评估仅依赖期望回报、忽视回报分布离散程度的问题,形式化并量化策略可复现性,提出采用带偏好参数的下置信界以平衡性能与可复现性,并通过实验验证其优势。

摘要:

强化学习(RL)中的许多应用通常在环境中存在噪声或随机性。除了对学习的影响之外,这些不确定性还会导致完全相同的策略在不同回合(roll-out)中表现不同,即产生不同的回报。RL中常见的评估程序仅使用期望回报来概括由此产生的回报分布,而没有考虑分布的离散程度。我们的工作将这种离散程度定义为策略可复现性:即策略在多次执行时获得相似性能的能力,这在某些实际应用中是一项关键属性。我们强调,仅使用期望回报的现有程序在两个方面存在局限:首先,无数具有广泛性能-可复现性权衡范围的回报分布可能具有相同的期望回报,这限制了其在比较策略时的有效性;其次,期望回报指标没有为从业者留下任何空间,使其无法为所考虑的应用选择最佳权衡值。在本工作中,我们通过推荐使用下置信界(Lower Confidence Bound)来解决这些局限,该指标取自贝叶斯优化,为用户提供一个偏好参数,以选择期望的性能-可复现性权衡。我们还对策略可复现性进行了形式化和量化,并通过在常见的不确定RL任务上对流行RL算法进行大量实验,证明了我们指标的优势。

英文摘要:

Many applications in Reinforcement Learning (RL) usually have noise or stochasticity present in the environment. Beyond their impact on learning, these uncertainties lead the exact same policy to perform differently, i.e. yield different return, from one roll-out to another. Common evaluation procedures in RL summarise the consequent return distributions using solely the expected return, which does not account for the spread of the distribution. Our work defines this spread as the policy reproducibility: the ability of a policy to obtain similar performance when rolled out many times, a crucial property in some real-world applications. We highlight that existing procedures that only use the expected return are limited on two fronts: first an infinite number of return distributions with a wide range of performance-reproducibility trade-offs can have the same expected return, limiting its effectiveness when used for comparing policies; second, the expected return metric does not leave any room for practitioners to choose the best trade-off value for considered applications. In this work, we address these limitations by recommending the use of Lower Confidence Bound, a metric taken from Bayesian optimisation that provides the user with a preference parameter to choose a desired performance-reproducibility trade-off. We also formalise and quantify policy reproducibility, and demonstrate the benefit of our metrics using extensive experiments of popular RL algorithms on common uncertain RL tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑