The exploration-exploitation trade-off is central to the description of adaptive behaviour in fields ranging from machine learning, to biology, to economics. While many approaches have been taken, one approach to solving this trade-off has been to equip or propose that agents possess an intrinsic 'exploratory drive' which is often implemented in terms of maximizing the agents information gain about the world -- an approach which has been widely studied in machine learning and cognitive science. In this paper we mathematically investigate the nature and meaning of such approaches and demonstrate that this combination of utility maximizing and information-seeking behaviour arises from the minimization of an entirely difference class of objectives we call divergence objectives. We propose a dichotomy in the objective functions underlying adaptive behaviour between \emph{evidence} objectives, which correspond to well-known reward or utility maximizing objectives in the literature, and \emph{divergence} objectives which instead seek to minimize the divergence between the agent's expected and desired futures, and argue that this new class of divergence objectives could form the mathematical foundation for a much richer understanding of the exploratory components of adaptive and intelligent action, beyond simply greedy utility maximization.
Provably Efficient Exploration in Reward Machines with Low Regret
Department of Computer Science, University of Copenhagen(哥本哈根大学计算机科学系);Department of Information and Communication Technologies, Universitat Pompeu Fabra(庞培法布拉大学信息与通信技术系);University of Lille(里尔大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);Centrale Lille(里尔中央理工学院)
Curious Causality-Seeking Agents Learn Meta Causal World
University of Bristol(布里斯托大学);Peking University(北京大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, Chinese Academy of Sciences(中国科学院人工智能学院);University College London(伦敦大学学院);King Abdullah University of Science and Technology(卡塔尔科技大学);The Swiss AI Lab, IDSIA-USI/SUPSI(瑞士AI实验室,IDSIA-USI/SUPSI)
Fidelity-based Probabilistic Q-learning for Control of Quantum Systems
Nanjing University(南京大学);Princeton University(普林斯顿大学);University of New South Wales(新南威尔士大学);Zhejiang University(浙江大学);City University of Hong Kong(香港城市大学);Washington University in St. Louis(圣路易斯华盛顿大学)
Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models
School of Mathematics, University of Edinburgh(爱丁堡大学数学学院);Alan Turing Institute(艾伦·图灵研究所);Department of Statistics, London School of Economics and Political Science(伦敦政治经济学院统计系)
Uncertainty-Aware Predictive Safety Filters for Probabilistic Neural Network Dynamics
Institute for Data Science in Mechanical Engineering (DSME), RWTH Aachen University(机械工程数据科学研究所(DSME),亚琛工业大学);Institute of Climate and Energy Systems (ICE), Energy Systems Engineering (ICE-1), Forschungszentrum Jülich GmbH(气候与能源系统研究所(ICE),能源系统工程(ICE-1),焦耳研究中心有限公司)