对未知的悲观主义激发保守主义
Pessimism About Unknown Unknowns Inspires Conservatism
摘要:
如果我们可以定义所有坏结果的集合,我们可以硬编码一个避免它们的代理;然而,在足够复杂的环境中,这是不可行的。我们不知道在文献中有任何通用的方法来避免新的故障模式。基于此,我们定义了一个理想化的贝叶斯强化学习器,它遵循一个策略,在一组世界模型上最大化最坏情况下的预期奖励。我们称这个代理为悲观的,因为它假设最坏的情况进行优化。标量参数通过改变考虑的世界模型集的大小来调整代理的悲观主义。我们的第一个主要贡献是:给定一个假设代理的模型类,一个足够悲观的代理不会导致“前所未有的事件”的概率$1-δ$,无论设计师知道如何精确地指定这些先例,他们所关心的。由于悲观主义不鼓励探索,在每个时间步,智能体可能会听从一个导师,他可能是一个人,也可能是我们想要改进的已知安全策略。我们的另一个主要贡献是,代理人的政策的价值至少接近导师,而推迟到导师的概率为0。在高风险环境中,我们可能希望先进的人工智能体谨慎地追求目标,即使允许智能体拥有任意的计算能力,这也是一个不小的问题;我们提出了一个正式的解决方案。
英文摘要:
If we could define the set of all bad outcomes, we could hard-code an agent which avoids them; however, in sufficiently complex environments, this is infeasible. We do not know of any general-purpose approaches in the literature to avoiding novel failure modes. Motivated by this, we define an idealized Bayesian reinforcement learner which follows a policy that maximizes the worst-case expected reward over a set of world-models. We call this agent pessimistic, since it optimizes assuming the worst case. A scalar parameter tunes the agent's pessimism by changing the size of the set of world-models taken into account. Our first main contribution is: given an assumption about the agent's model class, a sufficiently pessimistic agent does not cause "unprecedented events" with probability $1-δ$, whether or not designers know how to precisely specify those precedents they are concerned with. Since pessimism discourages exploration, at each timestep, the agent may defer to a mentor, who may be a human or some known-safe policy we would like to improve. Our other main contribution is that the agent's policy's value approaches at least that of the mentor, while the probability of deferring to the mentor goes to 0. In high-stakes environments, we might like advanced artificial agents to pursue goals cautiously, which is a non-trivial problem even if the agent were allowed arbitrary computing power; we present a formal solution.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。