Active inference is a first principle account of how autonomous agents operate in dynamic, non-stationary environments. This problem is also considered in reinforcement learning (RL), but limited work exists on comparing the two approaches on the same discrete-state environments. In this paper, we provide: 1) an accessible overview of the discrete-state formulation of active inference, highlighting natural behaviors in active inference that are generally engineered in RL; 2) an explicit discrete-state comparison between active inference and RL on an OpenAI gym baseline. We begin by providing a condensed overview of the active inference literature, in particular viewing the various natural behaviors of active inference agents through the lens of RL. We show that by operating in a pure belief-based setting, active inference agents can carry out epistemic exploration, and account for uncertainty about their environment in a Bayes-optimal fashion. Furthermore, we show that the reliance on an explicit reward signal in RL is removed in active inference, where reward can simply be treated as another observation; even in the total absence of rewards, agent behaviors are learned through preference learning. We make these properties explicit by showing two scenarios in which active inference agents can infer behaviors in reward-free environments compared to both Q-learning and Bayesian model-based RL agents; by placing zero prior preferences over rewards and by learning the prior preferences over the observations corresponding to reward. We conclude by noting that this formalism can be applied to more complex settings if appropriate generative models can be formulated. In short, we aim to demystify the behavior of active inference agents by presenting an accessible discrete state-space and time formulation, and demonstrate these behaviors in a OpenAI gym environment, alongside RL agents.
pymdp: A Python library for active inference in discrete state spaces
Max Planck Institute of Animal Behavior(马克斯·普朗克动物行为研究所);Centre for the Advanced Study of Collective Behaviour(集体行为高级研究中心);University of Konstanz(康斯坦茨大学);VERSES Research Lab(VERSES研究实验室);University of Oxford(牛津大学);Imperial College London(帝国理工学院);Northeastern University(东北大学);University College London(伦敦大学学院);University of Sussex(萨塞克斯大学)
Expanding the Active Inference Landscape: More Intrinsic Motivations in the Perception-Action Loop
Araya Inc.(Araya公司);Computational Creativity Group, Department of Computing, Goldsmiths, University of London(Goldsmiths大学计算创意小组);Game Innovation Lab, Department of Computer Science and Engineering, New York University(纽约大学游戏创新实验室);Sepia Lab, Adaptive Systems Research Group, Department of Computer Science, University of Hertfordshire(赫特福德大学计算机科学系Sepia实验室);Institute of Perception, Action and Behaviour, School of Informatics, The University of Edinburgh(爱丁堡大学信息学院感知、行为与行为研究所)
Sackler Centre for Consciousness Science(萨克勒意识科学中心);Evolutionary & Adaptive Systems Research Group(进化与自适应系统研究组);University of Sussex(萨塞克斯大学);University of Edinburgh(爱丁堡大学);Canadian Institute for Advanced Research(加拿大高级研究所)
The Phenomenology of Machine: A Comprehensive Analysis of the Sentience of the OpenAI-o1 Model Integrating Functionalism, Consciousness Theories, Active Inference, and AI Architectures
The Wellcome Centre for Human Neuroimaging(韦康比人类神经影像中心);UCL Queen Square Institute of Neurology(伦敦大学学院皇后广场神经学研究所);Imperial College London(伦敦帝国理工学院);University of Toronto(多伦多大学);Google Research(谷歌研究院);University of Amsterdam(阿姆斯特丹大学)
Institute of Cognitive Sciences and Technologies, National Research Council(国家研究委员会认知科学与技术研究所);Institute for High Performance Computing and Networking, National Research Council(国家研究委员会高性能计算与网络研究所);University College London(伦敦大学学院);VERSES AI Research Lab(VERSES人工智能研究实验室)
Belief-Space Control for Personalized Cancer Treatment via Active Inference
American Association for Cancer Research(美国癌症研究协会);AACR Project GENIE registry(AACR Project GENIE 注册中心);AACR Project GENIE Biopharma Collaborative(AACR Project GENIE 生物制药合作组织)
Mapping Husserlian phenomenology onto active inference
VERSES Research Lab(VERSES研究实验室);Université du Québec à Montréal(魁北克大学蒙特利尔分校);University of Oxford(牛津大学);University College London(伦敦大学学院);University of California, Merced(加利福尼亚大学默塞德分校)
Branching Time Active Inference: the theory and its generality
University of Kent(肯特大学);Imperial College London(帝国理工学院);Wellcome Centre for Human Neuroimaging, University College London(伦敦大学学院惠康人类神经影像中心);University of Birmingham(伯明翰大学)
Department of Electrical Engineering(电气工程系);Eindhoven University of Technology(埃因霍温理工大学);Eindhoven, the Netherlands(荷兰埃因霍温);Lazy Dynamics;Utrecht, the Netherlands(荷兰乌得勒支)
Active Inference in String Diagrams: A Categorical Account of Predictive Processing and Free Energy
Quantinuum(昆蒂努姆);Association for Mathematical Consciousness Science(数学意识科学协会);Ludwig Maximilian University of Munich(慕尼黑路德维希-马克西米利安大学);University of Bamberg(班贝格大学);Topos Institute(托波斯研究所);VERSES Research(VERSES研究院)
Branching Time Active Inference: empirical study and complexity class analysis
University of Kent(肯特大学);University of Birmingham(伯明翰大学)
主动推理是一种最先进的大脑建模框架,可以解释各种机制,如习惯形成,多巴胺能放电和好奇心。然而,最近的实现遭受的指数复杂度类时,计算所有可能的政策的时间范围内的先验。Fountas等人(2020)使用蒙特卡罗树搜索来解决这个问题,在两个不同的任务中得到了非常好的结果。此外,Champion等人(2021 a)提出了一种基于(时间)结构学习的树搜索方法。这是通过开发一种可变的消息传递方法来实现的,该方法可以实现贝叶斯网络的组合构建以进行主动推理。然而,这种消息传递树搜索方法,我们称之为分支时间主动推理(BTAI),从未进行过经验测试。在本文中,我们提出了一个实验研究的背景下,BTAI的迷宫解决代理。在这种情况下,我们表明,改进的先验偏好和更深入的搜索都有助于减轻局部最小值的脆弱性。然后,我们比较BTAI标准的主动推理(AcI)的图形导航任务。我们表明,对于小图,BTAI和AcI成功地解决了任务。对于较大的图,AcI表现出指数(空间)复杂性类,使该方法难以处理。然而,BTAI更有效地探索了政策空间,成功地扩展到更大的图表。然后,在冰冻湖环境下,将BTAI算法与POMCP算法进行了比较。实验表明,BTAI和POMCP算法积累了类似的奖励金额。此外,我们描述了当BTAI收到更多的奖励比POMCP代理,当相反的是真的。最后,我们将BTAI与Fountas et al(2020)在dSprite…
Designing explainable artificial intelligence with active inference: A framework for transparent introspection and decision-making
VERSES Research Lab(VERSES研究实验室);Université du Québec à Montréal(魁北克大学蒙特利尔分校);Humboldt-Universität zu Berlin(柏林洪堡大学);Macquarie University(麦考瑞大学);University College London(伦敦大学学院)