使用逐步相对可达性惩罚副作用
Penalizing side effects using stepwise relative reachability
摘要:
我们如何设计安全的强化学习代理,避免对其环境造成不必要的破坏?我们表明,目前的方法来惩罚副作用可以引入不良的激励措施,例如,以防止任何不可逆转的变化,在环境中,包括其他代理人的行动。为了隔离这种不良激励的来源,我们将副作用惩罚分为两个部分:基线状态和偏离此基线状态的度量。我们认为,这些激励措施中的一些来自基线的选择,和其他人产生的偏差措施的选择。我们引入了一个新的变种的逐步不作为基线和一个新的偏差测量的基础上相对可达的国家。这些设计选择的组合避免了给定的不期望的激励,而更简单的基线和不可达性度量失败了。我们证明了这一点,经验比较不同的组合的基线和偏差测量选择的一组gridworld实验,旨在说明可能的不良激励。
英文摘要:
How can we design safe reinforcement learning agents that avoid unnecessary disruptions to their environment? We show that current approaches to penalizing side effects can introduce bad incentives, e.g. to prevent any irreversible changes in the environment, including the actions of other agents. To isolate the source of such undesirable incentives, we break down side effects penalties into two components: a baseline state and a measure of deviation from this baseline state. We argue that some of these incentives arise from the choice of baseline, and others arise from the choice of deviation measure. We introduce a new variant of the stepwise inaction baseline and a new deviation measure based on relative reachability of states. The combination of these design choices avoids the given undesirable incentives, while simpler baselines and the unreachability measure fail. We demonstrate this empirically by comparing different combinations of baseline and deviation measure choices on a set of gridworld experiments designed to illustrate possible bad incentives.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。