Patch foraging is one of the most heavily studied behavioral optimization challenges in biology. However, despite its importance to biological intelligence, this behavioral optimization problem is understudied in artificial intelligence research. Patch foraging is especially amenable to study given that it has a known optimal solution, which may be difficult to discover given current techniques in deep reinforcement learning. Here, we investigate deep reinforcement learning agents in an ecological patch foraging task. For the first time, we show that machine learning agents can learn to patch forage adaptively in patterns similar to biological foragers, and approach optimal patch foraging behavior when accounting for temporal discounting. Finally, we show emergent internal dynamics in these agents that resemble single-cell recordings from foraging non-human primates, which complements experimental and theoretical work on the neural mechanisms of biological foraging. This work suggests that agents interacting in complex environments with ecologically valid pressures arrive at common solutions, suggesting the emergence of foundational computations behind adaptive, intelligent behavior in both biological and artificial agents.
World Models Unlock Optimal Foraging Strategies in Reinforcement Learning Agents
Center of Excellence in Analytics, Artificial Intelligence, and Information Governance, Bancolombia, Colombia（分析、人工智能和信息治理卓越中心，Bancolombia，哥伦比亚）；Department of Biomedical Engineering, Universidad de los Andes, Bogotá, Colombia（生物医学工程系， Universidad de los Andes，波哥大，哥伦比亚）；Department of Electrical and Electronics Engineering, Universidad de los Andes, Bogotá, Colombia（电气与电子工程系， Universidad de los Andes，波哥大，哥伦比亚）
Gazing at Rewards: Eye Movements as a Lens into Human and AI Decision-Making in Hybrid Visual Foraging
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Agency for Science, Technology and Research（新加坡科技研究局）；Harbin Institute of Technology（哈尔滨工业大学）；Beijing Institute of Technology（北京理工大学）；University of Virginia（弗吉尼亚大学）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）
Understanding an information forager's actions during interaction is very important for the study of interactive information retrieval. Although information spread in uncertain information space is substantially complex due to the high entanglement of users interacting with information objects~(text, image, etc.). However, an information forager, in general, accompanies a piece of information (information diet) while searching (or foraging) alternative contents, typically subject to decisive uncertainty. Such types of uncertainty are analogous to measurements in quantum mechanics which follow …
From Grunts to Lexicons: Emergent Language from Cooperative Foraging
Institute for Infocomm Research (I 2 {}^{\text{2}} R), A ∗ STAR, Singapore（新加坡信息与通信研究院）；College of Computing and Data Science, Nanyang Technological University, Singapore（南洋理工大学计算机科学与数据科学学院）；Department of Computer Science, The University of Manchester, United Kingdom（曼彻斯特大学计算机科学系）；Centre for Frontier AI Research (CFAR), A ∗ STAR, Singapore（前沿人工智能研究中心）；Sakana AI, Japan（日本Sakana AI）
Institute for Bioengineering of Catalonia (IBEC)（加泰罗尼亚生物工程研究所）；Universitat Pompeu Fabra (UPF)（庞培法布拉大学）；Catalan Institute for Research and Advanced Studies (ICREA)（加泰罗尼亚研究与高等研究院）
The sample-inefficiency problem in Artificial Intelligence refers to the inability of current Deep Reinforcement Learning models to optimize action policies within a small number of episodes. Recent studies have tried to overcome this limitation by adding memory systems and architectural biases to improve learning speed, such as in Episodic Reinforcement Learning. However, despite achieving incremental improvements, their performance is still not comparable to how humans learn behavioral policies. In this paper, we capitalize on the design principles of the Distributed Adaptive Control (DAC) t…
Shanghai Jiaotong University（上海交通大学）；Sea AI Lab（Sea人工智能实验室）
Visual imitation learning enables reinforcement learning agents to learn to behave from expert visual demonstrations such as videos or image sequences, without explicit, well-defined rewards. Previous research either adopted supervised learning techniques or induce simple and coarse scalar rewards from pixels, neglecting the dense information contained in the image demonstrations. In this work, we propose to measure the expertise of various local regions of image samples, or called \textit{patches}, and recover multi-dimensional \textit{patch rewards} accordingly. Patch reward is a more precis…
Mila – Quebec Artificial Intelligence Institute（魁北克人工智能研究所）；McGill University（麦吉尔大学）；Ottawa Hospital Research Institute（渥太华医院研究所在）；University of Ottawa（渥太华大学）
REFN: A Reinforcement-Learning-From-Network Framework against 1-day/n-day Exploitations
School of Artificial Intelligence, Hubei University（湖北大学人工智能学院）；Huazhong University of Science and Technology（华中科技大学）；University of Southern California（美国南加州大学）
Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning
Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Innovation Institute（上海创新研究院）
Real-time Digital Twinning of physical world scenes onto the Metaverse is necessary for a myriad of applications such as augmented-reality (AR) assisted driving. In AR assisted driving, physical environment scenes are first captured by Internet of Vehicles (IoVs) and are uploaded to the Metaverse. A central Metaverse Map Service Provider (MMSP) will aggregate information from all IoVs to develop a central Metaverse Map. Information from the Metaverse Map can then be downloaded into individual IoVs on demand and be delivered as AR scenes to the driver. However, the growing interest in developin…
Representing a scene and its constituent objects from raw sensory data is a core ability for enabling robots to interact with their environment. In this paper, we propose a novel approach for scene understanding, leveraging a hierarchical object-centric generative model that enables an agent to infer object category and pose in an allocentric reference frame using active inference, a neuro-inspired framework for action and perception. For evaluating the behavior of an active vision agent, we also propose a new benchmark where, given a target viewpoint of a particular object, the agent needs to…
Google（谷歌）；University of Oxford（牛津大学）；University of Cambridge（剑桥大学）；UC Berkeley（加州大学伯克利分校）；Columbia University（哥伦比亚大学）
There has recently been significant interest in training reinforcement learning (RL) agents in vision-based environments. This poses many challenges, such as high dimensionality and the potential for observational overfitting through spurious correlations. A promising approach to solve both of these problems is an attention bottleneck, which provides a simple and effective framework for learning high performing policies, even in the presence of distractions. However, due to poor scalability of attention architectures, these methods cannot be applied beyond low resolution visual inputs, using l…
The enactive approach to cognition is typically proposed as a viable alternative to traditional cognitive science. Enactive cognition displaces the explanatory focus from the internal representations of the agent to the direct sensorimotor interaction with its environment. In this paper, we investigate enactive learning through means of artificial agent simulations. We compare the performances of the enactive agent to an agent operating on classical reinforcement learning in foraging tasks within maze environments. The characteristics of the agents are analysed in terms of the accessibility of…
Tencent AI Lab（腾讯人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）
Whole-slide images (WSI) in computational pathology have high resolution with gigapixel size, but are generally with sparse regions of interest, which leads to weak diagnostic relevance and data inefficiency for each area in the slide. Most of the existing methods rely on a multiple instance learning framework that requires densely sampling local patches at high magnification. The limitation is evident in the application stage as the heavy computation for extracting patch-level features is inevitable. In this paper, we develop RLogist, a benchmarking deep reinforcement learning (DRL) method fo…
Reinforcement Learning (RL) can be considered as a sequence modeling task: given a sequence of past state-action-reward experiences, an agent predicts a sequence of next actions. In this work, we propose State-Action-Reward Transformer (StARformer) for visual RL, which explicitly models short-term state-action-reward representations (StAR-representations), essentially introducing a Markovian-like inductive bias to improve long-term modeling. Our approach first extracts StAR-representations by self-attending image state patches, action, and reward tokens within a short temporal window. These ar…
Sophisticated Learning: A novel algorithm for active learning during model-based planning
Laureate Institute for Brain Research, Tulsa, OK, USA（脑研究杰出研究院）；University of Cape Town, South Africa（开普敦大学）；MIND Institute（MIND研究院）；CSAM, University of the Witwatersrand, South Africa（CSAM，瓦特斯兰大学）；Delft University of Technology, Department of Cognitive Robotics（代尔夫特理工大学认知机器人学系）
本文提出 Sophisticated Learning (SL) 算法，将主动参数学习嵌入 Active Inference 的树搜索框架，通过反事实推理优化模型参数；实验表明，SL 在极端不确定性下显著提升了决策效率和收敛速度，并展现出优于 SI 和 BARL 的存活表现。
Vrije Universiteit Brussel（布鲁塞尔自由大学）；KU Leuven – University of Leuven（鲁汶大学）；Hasselt University（哈瑟尔特大学）
Epidemics of infectious diseases are an important threat to public health and global economies. Yet, the development of prevention strategies remains a challenging process, as epidemics are non-linear and complex processes. For this reason, we investigate a deep reinforcement learning approach to automatically learn prevention strategies in the context of pandemic influenza. Firstly, we construct a new epidemiological meta-population model, with 379 patches (one for each administrative district in Great Britain), that adequately captures the infection process of pandemic influenza. Our model b…
University of Leeds（利兹大学）；The Alan Turing Institute（艾伦·图灵研究所）；University of Glasgow（格拉斯哥大学）
Capturing and simulating intelligent adaptive behaviours within spatially explicit individual-based models remains an ongoing challenge for researchers. While an ever-increasing abundance of real-world behavioural data are collected, few approaches exist that can quantify and formalise key individual behaviours and how they change over space and time. Consequently, commonly used agent decision-making frameworks, such as event-condition-action rules, are often required to focus only on a narrow range of behaviours. We argue that these behavioural frameworks often do not reflect real-world scena…
Northeastern University（东北大学）；University of the Chinese Academy of Sciences（中国科学院大学）；China Medical University（中国医科大学）
The deep neural network is a research hotspot for histopathological image analysis, which can improve the efficiency and accuracy of diagnosis for pathologists or be used for disease screening. The whole slide pathological image can reach one gigapixel and contains abundant tissue feature information, which needs to be divided into a lot of patches in the training and inference stages. This will lead to a long convergence time and large memory consumption. Furthermore, well-annotated data sets are also in short supply in the field of digital pathology. Inspired by the pathologist's clinical di…
TextualRL: Textual Reinforcement Learning for Context Management
School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Shanghai Innovation Institute（上海创新研究院）；Fudan University（复旦大学）
Research School of Computer Science, ANU（澳大利亚国立大学计算机科学研究学院）；Google DeepMind（谷歌DeepMind）
The reinforcement learning (RL) framework formalizes the notion of learning with interactions. Many real-world problems have large state-spaces and/or action-spaces such as in Go, StarCraft, protein folding, and robotics or are non-Markovian, which cause significant challenges to RL algorithms. In this work we address the large action-space problem by sequentializing actions, which can reduce the action-space size significantly, even down to two actions at the expense of an increased planning horizon. We provide explicit and exact constructions and equivalence proofs for all quantities of inte…
Active Sensing with Meta-Reinforcement Learning for Emitter Localization from RF Observations
Fraunhofer Institute for Integrated Circuits IIS（弗劳恩霍夫集成电路研究所）；Positioning Systems Lab, University of Technology Nürnberg (UTN)（定位系统实验室，图恩大学）；Center for Artificial Intelligence, Technical University of Applied Sciences Würzburg-Schweinfurt（人工智能中心，应用技术大学魏玛-施魏尔堡）
PDiT: Interleaving Perception and Decision-making Transformers for Deep Reinforcement Learning
SenseTime Research（商汤科技研究院）；University of Cologne（科隆大学）；University of Chinese Academy of Sciences（中国科学院大学）；Gaoling School of AI, Renmin University of China（中国人民大学高瓴人工智能学院）；Peking University（北京大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Huazhong Agricultural University（华中农业大学）
Masked Autoencoders (MAEs) learn generalizable representations for image, text, audio, video, etc., by reconstructing masked input data from tokens of the visible data. Current MAE approaches for videos rely on random patch, tube, or frame-based masking strategies to select these tokens. This paper proposes AdaMAE, an adaptive masking strategy for MAEs that is end-to-end trainable. Our adaptive masking strategy samples visible tokens based on the semantic context using an auxiliary sampling network. This network estimates a categorical distribution over spacetime-patch tokens. The tokens that …
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Despite impressive results using reinforcement learning to solve complex problems from scratch, in robotics this has still been largely limited to model-based learning with very informative reward functions. One of the major challenges is that the reward landscape often has large patches with no gradient, making it difficult to sample gradients effectively. We show here that the robot state-initialization can have a more important effect on the reward landscape than is generally expected. In particular, we show the counter-intuitive benefit of including initializations that are unviable, in ot…
DeepMind（深度思维）；University of California Berkeley（加州大学伯克利分校）；Center for Human-Compatible AI（人类兼容人工智能中心）；Schwartz Reisman Institute for Technology and Society（施瓦茨·赖斯曼技术与社会研究所）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；OpenAI（开放人工智能研究中心）
How can societies learn to enforce and comply with social norms? Here we investigate the learning dynamics and emergence of compliance and enforcement of social norms in a foraging game, implemented in a multi-agent reinforcement learning setting. In this spatiotemporally extended game, individuals are incentivized to implement complex berry-foraging policies and punish transgressions against social taboos covering specific berry types. We show that agents benefit when eating poisonous berries is taboo, meaning the behavior is punished by other agents, as this helps overcome a credit-assignmen…
Self-Supervised Neuron Segmentation with Multi-Agent Reinforcement Learning
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
