Sophisticated Learning: A novel algorithm for active learning during model-based planning
Laureate Institute for Brain Research, Tulsa, OK, USA（脑研究杰出研究院）；University of Cape Town, South Africa（开普敦大学）；MIND Institute（MIND研究院）；CSAM, University of the Witwatersrand, South Africa（CSAM，瓦特斯兰大学）；Delft University of Technology, Department of Cognitive Robotics（代尔夫特理工大学认知机器人学系）
Rowan Hodson, Bruce Bassett, Charel van Hoof, Benjamin Rosman, Mark Solms, Jonathan P. Shock, Ryan Smith
本文提出 Sophisticated Learning (SL) 算法，将主动参数学习嵌入 Active Inference 的树搜索框架，通过反事实推理优化模型参数；实验表明，SL 在极端不确定性下显著提升了决策效率和收敛速度，并展现出优于 SI 和 BARL 的存活表现。
摘要：
我们引入了 Sophisticated Learning (SL)，这是一种规划学习算法，将主动参数学习嵌入到 Active Inference 的 Sophisticated Inference (SI) 树搜索框架中。与仅优化关于隐藏状态信念的 SI 不同，SL 还在每个模拟分支中更新关于模型参数的信念，从而能够对未来的观察将如何改善后续规划进行反事实推理。我们将 SL 与 Bayes-adaptive Reinforcement Learning (BARL) 智能体及其父算法 SI 进行了比较。使用一个生物学启发的季节性觅食任务（资源在 10x10 网格上概率性地转移），我们设计了迫使智能体在概率性奖励收集与信息收集之间取得平衡的实验。在需要快速学习的早期试验中，SL 智能体的存活时间平均比 SI 长 8.2%，比 Bayes-adaptive Reinforcement Learning 长 35%。虽然 SL 和 SI 表现出相同的收敛性能，但 SL 达到这种收敛的速度比 SI 快 40%。此外，在改变的环境配置中，SL 表现出对其他算法的稳健优势。我们的结果表明，将主动学习纳入多步规划实质性地改善了极端不确定性下的决策，并进一步证实了 Active Inference 在建模生物学相关行为方面的广泛效用。
英文摘要：
We introduce Sophisticated Learning (SL), a planning-to-learn algorithm that embeds active parameter learning inside the Sophisticated Inference (SI) tree-search framework of Active Inference. Unlike SI -- which optimizes beliefs about hidden states -- SL also updates beliefs about model parameters within each simulated branch, enabling counterfactual reasoning about how future observations would improve subsequent planning.
We compared SL with Bayes-adaptive Reinforcement Learning (BARL) agents as well as with its parent algorithm, SI. Using a biologically inspired seasonal foraging task in which resources shift probabilistically over a 10x10 grid, we designed experiments that forced agents to balance probabilistic reward harvesting against information gathering.
In early trials, where rapid learning is vital, SL agents survive, on average, 8.2% longer than SI and 35% longer than Bayes-adaptive Reinforcement Learning. While both SL and SI showed equal convergence performance, SL reached this convergence 40% faster than SI. Additionally, SL showed robust out-performance of other algorithms in altered environment configurations.
Our results show that incorporating active learning into multi-step planning materially improves decision making under radical uncertainty, and reinforces the broader utility of Active Inference for modeling biologically relevant behavior.
The Wellcome Centre for Human Neuroimaging（韦康比人类神经影像中心）；UCL Queen Square Institute of Neurology（伦敦大学学院皇后广场神经学研究所）；Imperial College London（伦敦帝国理工学院）；University of Toronto（多伦多大学）；Google Research（谷歌研究院）；University of Amsterdam（阿姆斯特丹大学）
Active inference offers a first principle account of sentient behaviour, from which special and important cases can be derived, e.g., reinforcement learning, active learning, Bayes optimal inference, Bayes optimal design, etc. Active inference resolves the exploitation-exploration dilemma in relation to prior preferences, by placing information gain on the same footing as reward or value. In brief, active inference replaces value functions with functionals of (Bayesian) beliefs, in the form of an expected (variational) free energy. In this paper, we consider a sophisticated kind of active infe…
MIT（麻省理工学院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Width-based planning has shown promising results on Atari 2600 games using pixel input, while using substantially fewer environment interactions than reinforcement learning. Recent width-based approaches have computed feature vectors for each screen using a hand designed feature set or a variational autoencoder trained on game screens (VAE-IW), and prune screens that do not have novel features during the search. We propose Olive (Online-VAE-IW), which updates the VAE features online using active learning to maximize the utility of screens observed during planning. Experimental results in 55 At…
Okinawa Institute of Science and Technology（冲绳科学技术大学院大学）
What is the difference between goal-directed and habitual behavior? We propose a novel computational framework of decision making with Bayesian inference, in which everything is integrated as an entire neural network model. The model learns to predict environmental state transitions by self-exploration and generating motor actions by sampling stochastic internal states ${z}$. Habitual behavior, which is obtained from the prior distribution of ${z}$, is acquired by reinforcement learning. Goal-directed behavior is determined from the posterior distribution of ${z}$ by planning, using active inf…
Attention control is a key cognitive ability for humans to select information relevant to the current task. This paper develops a computational model of attention and an algorithm for attention-based probabilistic planning in Markov decision processes. In attention-based planning, the robot decides to be in different attention modes. An attention mode corresponds to a subset of state variables monitored by the robot. By switching between different attention modes, the robot actively perceives task-relevant information to reduce the cost of information acquisition and processing, while achievin…
Aerial robots are increasingly being utilized for environmental monitoring and exploration. However, a key challenge is efficiently planning paths to maximize the information value of acquired data as an initially unknown environment is explored. To address this, we propose a new approach for informative path planning based on deep reinforcement learning (RL). Combining recent advances in RL and robotic applications, our method combines tree search with an offline-learned neural network predicting informative sensing actions. We introduce several components making our approach applicable for r…
University of Amsterdam（阿姆斯特丹大学）；University of Oxford（牛津大学）；University of Liverpool（利物浦大学）；Delft University of Technology（代尔夫特理工大学）
In active perception tasks, an agent aims to select sensory actions that reduce its uncertainty about one or more hidden variables. While partially observable Markov decision processes (POMDPs) provide a natural model for such problems, reward functions that directly penalize uncertainty in the agent's belief can remove the piecewise-linear and convex property of the value function required by most POMDP planners. Furthermore, as the number of sensors available to the agent grows, the computational cost of POMDP planning grows exponentially with it, making POMDP planning infeasible with tradit…
Microsoft Research（微软研究院）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Carnegie Mellon University（卡内基梅隆大学）
Training task-completion dialogue agents with reinforcement learning usually requires a large number of real user experiences. The Dyna-Q algorithm extends Q-learning by integrating a world model, and thus can effectively boost training efficiency using simulated experiences generated by the world model. The effectiveness of Dyna-Q, however, depends on the quality of the world model - or implicitly, the pre-specified ratio of real vs. simulated experiences used for Q-learning. To this end, we extend the recently proposed Deep Dyna-Q (DDQ) framework by integrating a switcher that automatically …
Institute of Cognitive Sciences and Technologies, National Research Council（国家研究委员会认知科学与技术研究所）；Institute for High Performance Computing and Networking, National Research Council（国家研究委员会高性能计算与网络研究所）；University College London（伦敦大学学院）；VERSES AI Research Lab（VERSES人工智能研究实验室）
Centre for Artificial Intelligence Research（人工智能研究中心）；University of KwaZulu-Natal（夸祖鲁-纳塔尔大学）；CSIR Meraka（南非科学与工业研究理事会梅拉卡研究所）
This article presents an agent architecture for controlling an autonomous agent in stochastic environments. The architecture combines the partially observable Markov decision process (POMDP) model with the belief-desire-intention (BDI) framework. The Hybrid POMDP-BDI agent architecture takes the best features from the two approaches, that is, the online generation of reward-maximizing courses of action from POMDP theory, and sophisticated multiple goal management from BDI theory. We introduce the advances made since the introduction of the basic architecture, including (i) the ability to pursu…
Lincoln Institute for Agri-food Technology, University of Lincoln（林肯大学农业食品技术林肯研究所）
In our previous work, we designed a systematic policy to prioritize sampling locations to lead significant accuracy improvement in spatial interpolation by using the prediction uncertainty of Gaussian Process Regression (GPR) as "attraction force" to deployed robots in path planning. Although the integration with Traveling Salesman Problem (TSP) solvers was also shown to produce relatively short travel distance, we here hypothesise several factors that could decrease the overall prediction precision as well because sub-optimal locations may eventually be included in their paths. To address thi…
MRC Brain Networks Dynamics Unit（英国医学研究理事会脑网络动力学组）；University of Oxford（牛津大学）；Verses Research Lab（Verses研究实验室）；University of Sussex（萨塞克斯大学）
Recent work has uncovered close links between between classical reinforcement learning algorithms, Bayesian filtering, and Active Inference which lets us understand value functions in terms of Bayesian posteriors. An alternative, but less explored, model-free RL algorithm is the successor representation, which expresses the value function in terms of a successor matrix of expected future state occupancies. In this paper, we derive the probabilistic interpretation of the successor representation in terms of Bayesian filtering and thus design a novel active inference agent architecture utilizing…
TourPlanner: A Competitive Consensus Framework with Constraint-Gated Reinforcement Learning for Travel Planning
Xiaohongshu Inc.（小红书公司）；Renmin University of China（中国人民大学）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS）
IITB-Monash Research Academy（印度理工学院孟买分校-莫纳什研究学院）；IIT Bombay（印度理工学院孟买分校）；Monash University（莫纳什大学）；UCL（伦敦大学学院）；CIFAR（加拿大高等研究院）
Active inference has emerged as an alternative approach to control problems given its intuitive (probabilistic) formalism. However, despite its theoretical utility, computational implementations have largely been restricted to low-dimensional, deterministic settings. This paper highlights that this is a consequence of the inability to adequately model stochastic transition dynamics, particularly when an extensive policy (i.e., action trajectory) space must be evaluated during planning. Fortunately, recent advancements propose a modified planning algorithm for finite temporal horizons. We build…
Seoul National University（首尔国立大学）；Ewha Womans University（梨花女子大学）；Pukyong National University（釜庆国立大学）
Energy management systems (EMS) are becoming increasingly important in order to utilize the continuously growing curtailed renewable energy. Promising energy storage systems (ESS), such as batteries and green hydrogen should be employed to maximize the efficiency of energy stakeholders. However, optimal decision-making, i.e., planning the leveraging between different strategies, is confronted with the complexity and uncertainties of large-scale problems. Here, we propose a sophisticated deep reinforcement learning (DRL) methodology with a policy-based algorithm to realize the real-time optimal…
University of Pennsylvania（宾夕法尼亚大学）；Cornell University（康奈尔大学）
In this paper, we propose a novel Reinforcement Learning approach for solving the Active Information Acquisition problem, which requires an agent to choose a sequence of actions in order to acquire information about a process of interest using on-board sensors. The classic challenges in the information acquisition problem are the dependence of a planning algorithm on known models and the difficulty of computing information-theoretic cost functions over arbitrary distributions. In contrast, the proposed framework of reinforcement learning does not require any knowledge on models and alleviates …
Active inference is a process theory of the brain that states that all living organisms infer actions in order to minimize their (expected) free energy. However, current experiments are limited to predefined, often discrete, state spaces. In this paper we use recent advances in deep learning to learn the state space and approximate the necessary probability distributions to engage in active inference.
University of Texas, Dallas（得克萨斯大学达拉斯分校）；Indiana University, Bloomington（印第安纳大学伯明顿分校）；Washington State University（华盛顿州立大学）；University of Pennsylvania（宾夕法尼亚大学）
Planning with preferences has been employed extensively to quickly generate high-quality plans. However, it may be difficult for the human expert to supply this information without knowledge of the reasoning employed by the planner and the distribution of planning problems. We consider the problem of actively eliciting preferences from a human expert during the planning process. Specifically, we study this problem in the context of the Hierarchical Task Network (HTN) planning framework as it allows easy interaction with the human. Our experimental results on several diverse planning domains sh…
On Predictive planning and counterfactual learning in active inference
Turner Institute for Brain and Mental Health（特纳脑与心理健康研究所）；School of Psychological Sciences（心理科学学院）；Monash University（莫纳什大学）；IITB-Monash Research Academy（印度理工学院孟买分校-莫纳什大学研究院）；IIT Bombay（印度理工学院孟买分校）；RIKEN Center for Brain Science（理化学研究所脑科学中心）；University College London（伦敦大学学院）；Wellcome Trust Centre for Human Neuroimaging（惠康信托人类神经影像中心）；CIFAR（加拿大高级研究所）
Active sensing and planning in unknown, cluttered environments is an open challenge for robots intending to provide home service, search and rescue, narrow-passage inspection, and medical assistance. Although many active sensing methods exist, they often consider open spaces, assume known settings, or mostly do not generalize to real-world scenarios. We present the active neural sensing approach that generates the kinematically feasible viewpoint sequences for the robot manipulator with an in-hand camera to gather the minimum number of observations needed to reconstruct the underlying environm…
Turner Institute for Brain and Mental Health, Monash University（莫纳什大学特纳脑与心理健康研究所）；IITB-Monash Research Academy（印度理工学院孟买分校-莫纳什大学研究院）；Department of Electrical Engineering, IIT Bombay（印度理工学院孟买分校电气工程系）；Wellcome Centre for Human Neuroimaging, University College London（伦敦大学学院惠康人类神经影像中心）；Department of Mathematics, Imperial College London（伦敦帝国理工学院数学系）；VERSES Research Lab（VERSES研究实验室）；CIFAR（加拿大高级研究所）
Department of Electrical Engineering（电气工程系）；Eindhoven University of Technology（埃因霍温理工大学）；Eindhoven, the Netherlands（荷兰埃因霍温）；Lazy Dynamics；Utrecht, the Netherlands（荷兰乌得勒支）
Interactively Robot Action Planning with Uncertainty Analysis and Active Questioning by Large Language Model
Waseda University（早稻田大学）；Fujitsu Limited（富士通有限公司）；Waseda Research Institute for Science and Engineering（早稻田大学科学与工程研究院）；National Institute of Advanced Industrial Science and Technology（日本产业技术综合研究所）
National University of Singapore（新加坡国立大学）；Massachusetts Institute of Technology（麻省理工学院）
This paper presents a novel nonmyopic adaptive Gaussian process planning (GPP) framework endowed with a general class of Lipschitz continuous reward functions that can unify some active learning/sensing and Bayesian optimization criteria and offer practitioners some flexibility to specify their desired choices for defining new tasks/problems. In particular, it utilizes a principled Bayesian sequential decision problem framework for jointly and naturally optimizing the exploration-exploitation trade-off. In general, the resulting induced GPP policy cannot be derived exactly due to an uncountabl…
The objective of this work is to augment the basic abilities of a robot by learning to use new sensorimotor primitives to enable the solution of complex long-horizon problems. Solving long-horizon problems in complex domains requires flexible generative planning that can combine primitive abilities in novel combinations to solve problems as they arise in the world. In order to plan to combine primitive actions, we must have models of the preconditions and effects of those actions: under what circumstances will executing this primitive achieve some particular effect in the world? We use, and de…
We propose a hybrid combination of active inference and behavior trees (BTs) for reactive action planning and execution in dynamic environments, showing how robotic tasks can be formulated as a free-energy minimization problem. The proposed approach allows handling partially observable initial states and improves the robustness of classical BTs against unexpected contingencies while at the same time reducing the number of nodes in a tree. In this work, we specify the nominal behavior offline, through BTs. However, in contrast to previous approaches, we introduce a new type of leaf node to spec…
MINT: Minimal Information Neuro-Symbolic Tree for Objective-Driven Knowledge-Gap Reasoning and Active Elicitation
Department of Electrical and Computer Engineering, George Washington University（乔治华盛顿大学电气与计算机工程系）；Department of Electrical and Computer Engineering, Northeastern University（东北大学电气与计算机工程系）
Okinawa Institute of Science and Technology（冲绳科学技术大学院大学）
It is crucial to ask how agents can achieve goals by generating action plans using only partial models of the world acquired through habituated sensory-motor experiences. Although many existing robotics studies use a forward model framework, there are generalization issues with high degrees of freedom. The current study shows that the predictive coding (PC) and active inference (AIF) frameworks, which employ a generative model, can develop better generalization by learning a prior distribution in a low dimensional latent state space representing probabilistic structures extracted from well hab…
Okinawa Institute of Science and Technology（冲绳科学技术大学院大学）
We show that goal-directed action planning and generation in a teleological framework can be formulated using the free energy principle. The proposed model, which is built on a variational recurrent neural network model, is characterized by three essential features. These are that (1) goals can be specified for both static sensory states, e.g., for goal images to be reached and dynamic processes, e.g., for moving around an object, (2) the model can not only generate goal-directed action plans, but can also understand goals by sensory observation, and (3) the model generates future action plans…
The Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；School of Computer Science and Engineering, University of Washington（华盛顿大学计算机科学与工程学院）
Robots operate in environments with varying implicit structure. For instance, a helicopter flying over terrain encounters a very different arrangement of obstacles than a robotic arm manipulating objects on a cluttered table top. State-of-the-art motion planning systems do not exploit this structure, thereby expending valuable planning effort searching for implausible solutions. We are interested in planning algorithms that actively infer the underlying structure of the valid configuration space during planning in order to find solutions with minimal effort. Consider the problem of evaluating …
KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents
Zhejiang University（浙江大学）；ZJU-Ant Group Joint Research Center for Knowledge Graphs（浙江大学-蚂蚁集团知识图谱联合研究中心）；Ant Group（蚂蚁集团）；National University of Singapore（新加坡国立大学）
