Since its inception, artificial intelligence has relied upon a theoretical foundation centered around perfect rationality as the desired property of intelligent systems. We argue, as others have done, that this foundation is inadequate because it imposes fundamentally unsatisfiable requirements. As a result, there has arisen a wide gap between theory and practice in AI, hindering progress in the field. We propose instead a property called bounded optimality. Roughly speaking, an agent is bounded-optimal if its program is a solution to the constrained optimization problem presented by its architecture and the task environment. We show how to construct agents with this property for a simple class of machine architectures in a broad class of real-time environments. We illustrate these results using a simple model of an automated mail sorting facility. We also define a weaker property, asymptotic bounded optimality (ABO), that generalizes the notion of optimality in classical complexity theory. We then construct universal ABO programs, i.e., programs that are ABO no matter what real-time constraints are applied. Universal ABO programs can be used as building blocks for more complex systems. We conclude with a discussion of the prospects for bounded optimality as a theoretical basis for AI, and relate it to similar trends in philosophy, economics, and game theory.
Provably Efficient Reinforcement Learning in Decentralized General-Sum Markov Games
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
本文讨论了通过分散式多智能体强化学习有效学习一般和马尔可夫博弈均衡的问题。考虑到计算纳什均衡(NE)的根本困难,我们的目标是找到一个粗略的相关均衡(CCE),一个解决方案的概念,通过允许可能的代理策略之间的相关性来推广NE。我们提出了一种算法,其中每个代理独立运行乐观的V-学习(Q-学习的变体),以有效地探索未知的环境,同时使用稳定的在线镜像下降(OMD)子例程进行策略更新。我们发现,代理可以找到一个$ε$ -近似CCE在最多$\widetilde{O}( H^6S A /ε^2)$情节,其中$S$是状态的数量,$A$是最大的个人行动空间的大小,和$H$是情节的长度。这似乎是通用一般和马尔可夫博弈学习的第一个样本复杂性结果。我们的研究结果依赖于一个新的调查的任何时候都高概率的遗憾界的OMD与动态学习率和加权遗憾,这将是独立的利益。我们的算法的一个关键特征是,它是完全\emph{decentralized},在这个意义上说,每个代理只能访问其本地信息,是完全不知道其他人的存在。这样,我们的算法可以很容易地扩展到任意数量的代理,而不会受到代理数量的指数依赖。
Negotiated Reasoning: On Provably Addressing Relative Over-Generalization
East China Normal University(华东师范大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳人工智能与机器人研究院);Tongji University(同济大学)
Federal Research Center “Computer Science and Control” of Russian Academy of Sciences(俄罗斯科学院计算机科学与控制联邦研究中心);Peoples’ Friendship University of Russia (RUDN University)(俄罗斯人民友谊大学(RUDN大学));National Research University “Higher School of Economics”(国家研究型高等经济大学);Ben-Gurion University of the Negev(内盖夫本-古里安大学)
Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning
School of Cyber Science and Engineering, Huazhong University of Science and Technology(华中科技大学网络空间科学与工程学院);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学)
Performance of Bounded-Rational Agents With the Ability to Self-Modify
Basic Algorithms Research Copenhagen (BARC)(哥本哈根基础算法研究院);Univ. of Copenhagen(哥本哈根大学);University of Oxford(牛津大学)
嵌入在复杂环境中的代理的自我修改是难以避免的,无论是通过直接方式(例如自己的代码修改)还是间接方式(例如影响操作员,利用错误或环境)。有人认为,智能代理有动机避免修改其效用函数,使其未来的实例朝着相同的目标工作。Everitt et al.(2016)正式表明,提供自我修改的选项对于完全理性的代理人来说是无害的。我们表明,这一结果不再是真实的代理有限理性。在这样的代理中,自我修改可能导致性能的指数恶化和先前对准的代理的逐渐失准。我们研究了这种效应的大小如何取决于代理人理性中缺陷的类型和大小(下面1-4)。我们还讨论了模型假设和更广泛的问题和框架空间。我们研究了代理可以被限制理性的四种方式:它要么(1)不总是选择最佳行动,(2)与人类价值观不完全一致,(3)具有不准确的环境模型,或者(4)使用错误的时间贴现因子。我们表明,虽然在情况(2)-(4)中,由代理的不完美性引起的不对准不会随着时间的推移而增加,但(1)不对准可能呈指数增长。
A maximum entropy model of bounded rational decision-making with prior beliefs and market feedback
Centre for Complex Systems, The University of Sydney(悉尼大学复杂系统中心);Securities Industry Research Centre of Asia-Pacific(亚太证券业研究中心);CoreLogic, Inc.(CoreLogic公司)