通过考虑未来的任务来避免副作用
Avoiding Side Effects By Considering Future Tasks
摘要:
设计奖励函数是困难的:设计者必须指定做什么(完成任务意味着什么)以及不做什么(完成任务时应该避免的副作用)。为了减轻奖励设计者的负担,我们提出了一个算法来自动生成一个辅助奖励函数,惩罚副作用。此辅助目标奖励完成未来可能任务的能力,如果代理在当前任务期间引起副作用,则该能力会降低。未来任务奖励还可以激励智能体干扰环境中使未来任务更难实现的事件,例如其他智能体的不可逆动作。为了避免这种干扰激励,我们引入了一个基线策略,它代表了一个默认的行动过程(比如什么都不做),并使用它来过滤掉默认情况下无法实现的未来任务。我们正式定义干扰激励,并表明,未来的任务方法与基线政策,避免这些激励在确定性的情况下。使用测试副作用和干扰的网格世界环境,我们表明,我们的方法避免了干扰,并且比惩罚不可逆行为的常见方法更有效地避免副作用。
英文摘要:
Designing reward functions is difficult: the designer has to specify what to do (what it means to complete the task) as well as what not to do (side effects that should be avoided while completing the task). To alleviate the burden on the reward designer, we propose an algorithm to automatically generate an auxiliary reward function that penalizes side effects. This auxiliary objective rewards the ability to complete possible future tasks, which decreases if the agent causes side effects during the current task. The future task reward can also give the agent an incentive to interfere with events in the environment that make future tasks less achievable, such as irreversible actions by other agents. To avoid this interference incentive, we introduce a baseline policy that represents a default course of action (such as doing nothing), and use it to filter out future tasks that are not achievable by default. We formally define interference incentives and show that the future task approach with a baseline policy avoids these incentives in the deterministic case. Using gridworld environments that test for side effects and interference, we show that our method avoids interference and is more effective for avoiding side effects than the common approach of penalizing irreversible actions.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。