我提出反对Silver、Singh、Precup和Sutton（https://www.sciencedirect.com/science/article/pii/S0004370221000862）所提出的假设的论据：奖励最大化不足以解释与自然智能和人工智能相关的许多活动，包括知识、学习、感知、社会智能、进化、语言、泛化和模仿。我表明这种“以盈利为目的的还原”（reductio ad lucrum）在知识渊源上可追溯至“经济人”（Homo economicus）的政治经济学，并与激进版本的行为主义有实质重叠。我论证了为什么强化学习范式，尽管在某些实际应用中显示出实用性，却是一个不完整的智能框架——无论是对于自然智能还是人工智能。智能行为的复杂性并非简单地在奖励最大化之上的二阶复杂化。这一事实对开发实用、智能、安全且稳健的人工智能体具有深远影响。
英文摘要：
I present arguments against the hypothesis put forward by Silver, Singh, Precup, and Sutton ( https://www.sciencedirect.com/science/article/pii/S0004370221000862 ) : reward maximization is not enough to explain many activities associated with natural and artificial intelligence including knowledge, learning, perception, social intelligence, evolution, language, generalisation and imitation. I show such reductio ad lucrum has its intellectual origins in the political economy of Homo economicus and substantially overlaps with the radical version of behaviourism. I show why the reinforcement learning paradigm, despite its demonstrable usefulness in some practical application, is an incomplete framework for intelligence -- natural and artificial. Complexities of intelligent behaviour are not simply second-order complications on top of reward maximisation. This fact has profound implications for the development of practically usable, smart, safe and robust artificially intelligent agents.
In most common settings of Markov Decision Process (MDP), an agent evaluate a policy based on expectation of (discounted) sum of rewards. However in many applications this criterion might not be suitable from two perspective: first, in risk aversion situation expectation of accumulated rewards is not robust enough, this is the case when distribution of accumulated reward is heavily skewed; another issue is that many applications naturally take several objective into consideration when evaluating a policy, for instance in autonomous driving an agent needs to balance speed and safety when choosi…
Future of Humanity Institute（人类未来研究所）；University of Oxford（牛津大学）；Vector Institute（向量研究所）；University of Toronto（多伦多大学）；Machine Intelligence Research Institute（机器智能研究所）
Inverse reinforcement learning (IRL) attempts to infer human rewards or preferences from observed behavior. Since human planning systematically deviates from rationality, several approaches have been tried to account for specific human shortcomings. However, the general problem of inferring the reward function of an agent of unknown rationality has received little attention. Unlike the well-known ambiguity problems in IRL, this one is practically relevant but cannot be resolved by observing the agent's policy in enough environments. This paper shows (1) that a No Free Lunch result implies it i…
Center for Translational Neuroscience, University of Oregon（俄勒冈大学转化神经科学中心）；Universidade do Vale do Rio dos Sinos（淡水河谷大学）；AI Lab, Vrije Universiteit Brussel（布鲁塞尔自由大学人工智能实验室）；Vrije Universiteit Brussel（布鲁塞尔自由大学）；HU University of Applied Sciences Utrecht（乌得勒支应用科学大学）；Universiteit Hasselt（哈瑟尔特大学）；Katholieke Universiteit Leuven（鲁汶天主教大学）；Deakin University（迪肯大学）；Federation University Australia（澳大利亚联邦大学）；Linköping University（林雪平大学）；National University of Ireland Galway（爱尔兰国立高威大学）
The recent paper `"Reward is Enough" by Silver, Singh, Precup and Sutton posits that the concept of reward maximisation is sufficient to underpin all intelligence, both natural and artificial. We contest the underlying assumption of Silver et al. that such reward can be scalar-valued. In this paper we explain why scalar rewards are insufficient to account for some aspects of both biological and computational intelligence, and argue in favour of explicitly multi-objective models of reward maximisation. Furthermore, we contend that even if scalar reward functions can trigger intelligent behaviou…
Reinforcement Learning has suffered from poor reward specification, and issues for reward hacking even in simple enough domains. Preference Based Reinforcement Learning attempts to solve the issue by utilizing binary feedbacks on queried trajectory pairs by a human in the loop indicating their preferences about the agent's behavior to learn a reward model. In this work, we present a state augmentation technique that allows the agent's reward model to be robust and follow an invariance consistency that significantly improved performance, i.e. the reward recovery and subsequent return computed u…
Reinforcement Learning (RL) has been witnessed its potential for training a dialogue policy agent towards maximizing the accumulated rewards given from users. However, the reward can be very sparse for it is usually only provided at the end of a dialog session, which causes unaffordable interaction requirements for an acceptable dialog agent. Distinguished from many efforts dedicated to optimizing the policy and recovering the reward alternatively which suffers from easily getting stuck in local optima and model collapse, we decompose the adversarial training into two steps: 1) we integrate a …
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
People's decisions about how to allocate their limited computational resources are essential to human intelligence. An important component of this metacognitive ability is deciding whether to continue thinking about what to do and move on to the next decision. Here, we show that people acquire this ability through learning and reverse-engineer the underlying learning mechanisms. Using a process-tracing paradigm that externalises human planning, we find that people quickly adapt how much planning they perform to the cost and benefit of planning. To discover the underlying metacognitive learning…
Learning to Decide with Just Enough: Information-Theoretic Context Summarization for CMDPs
Sichuan University（四川大学）；University of Toronto（多伦多大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）；University of Arizona（亚利桑那大学）
Google Brain（谷歌大脑）；UC Berkeley（加州大学伯克利分校）；Stanford（斯坦福大学）；Google AI Resident（谷歌AI驻场研究员）
Imitation learning allows agents to learn complex behaviors from demonstrations. However, learning a complex vision-based task may require an impractical number of demonstrations. Meta-imitation learning is a promising approach towards enabling agents to learn a new task from one or a few demonstrations by leveraging experience from learning similar tasks. In the presence of task ambiguity or unobserved dynamics, demonstrations alone may not provide enough information; an agent must also try the task to successfully infer a policy. In this work, we propose a method that can learn to learn from…
The U.S. Securities and Exchange Commission（美国证券交易委员会）
Legg and Hutter, as well as subsequent authors, considered intelligent agents through the lens of interaction with reward-giving environments, attempting to assign numeric intelligence measures to such agents, with the guiding principle that a more intelligent agent should gain higher rewards from environments in some aggregate sense. In this paper, we consider a related question: rather than measure numeric intelligence of one Legg- Hutter agent, how can we compare the relative intelligence of two Legg-Hutter agents? We propose an elegant answer based on the following insight: we can view Leg…
Conveying complex objectives to reinforcement learning (RL) agents can often be difficult, involving meticulous design of reward functions that are sufficiently informative yet easy enough to provide. Human-in-the-loop RL methods allow practitioners to instead interactively teach agents through tailored feedback; however, such approaches have been challenging to scale since human feedback is very expensive. In this work, we aim to make this process more sample- and feedback-efficient. We present an off-policy, interactive RL algorithm that capitalizes on the strengths of both feedback and off-…
A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models
Generative AI Lab, College of Computing and Data Science, Nanyang Technological University, Singapore（生成式人工智能实验室，计算与数据科学学院，南洋理工大学，新加坡）；Tongyi Lab, Alibaba Group（通义实验室，阿里巴巴集团）
Reinforcement Learning (RL) is a general framework concerned with an agent that seeks to maximize rewards in an environment. The learning typically happens through trial and error using explorative methods, such as epsilon-greedy. There are two approaches, model-based and model-free reinforcement learning, that show concrete results in several disciplines. Model-based RL learns a model of the environment for learning the policy while model-free approaches are fully explorative and exploitative without considering the underlying environment dynamics. Model-free RL works conceptually well in sim…
Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）；Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen)（深圳市未来智联网络研究院）
Your Models Have Thought Enough: Training Large Reasoning Models to Stop Overthinking
Shanghai Institute of Artificial Intelligence for Education, East China Normal University（华东师范大学上海智能教育研究院）；School of Data Science, Fudan University（复旦大学数据科学学院）；College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院）；Antgroup（蚂蚁集团）
Center for Theoretical Neuroscience, Columbia University（哥伦比亚大学理论神经科学中心）；IBM Thomas J. Watson Research Center（IBM托马斯·J·沃森研究中心）；Mila - Quebec AI Institute, University of Montreal（蒙特利尔大学Mila魁北克人工智能研究所）
Artificial behavioral agents are often evaluated based on their consistent behaviors and performance to take sequential actions in an environment to maximize some notion of cumulative reward. However, human decision making in real life usually involves different strategies and behavioral trajectories that lead to the same empirical outcome. Motivated by clinical literature of a wide range of neurological and psychiatric disorders, we propose here a more general and flexible parametric framework for sequential decision making that involves a two-stream reward processing mechanism. We demonstrat…
In this paper, we propose a model-free reinforcement learning method to synthesize control policies for motion planning problems with continuous states and actions. The robot is modelled as a labeled discrete-time Markov decision process (MDP) with continuous state and action spaces. Linear temporal logics (LTL) are used to specify high-level tasks. We then train deep neural networks to approximate the value function and policy using an actor-critic reinforcement learning method. The LTL specification is converted into an annotated limit-deterministic Büchi automaton (LDBA) for continuously sh…
Groundbreaking successes have been achieved by Deep Reinforcement Learning (DRL) in solving practical decision-making problems. Robotics, in particular, can involve high-cost hardware and human interactions. Hence, scrupulous evaluations of trained models are required to avoid unsafe behaviours in the operational environment. However, designing metrics to measure the safety of a neural network is an open problem, since standard evaluation parameters (e.g., total reward) are not informative enough. In this paper, we present a semi-formal verification approach for decision-making tasks, based on…
Designing agents that acquire knowledge autonomously and use it to solve new tasks efficiently is an important challenge in reinforcement learning. Knowledge acquired during an unsupervised pre-training phase is often transferred by fine-tuning neural network weights once rewards are exposed, as is common practice in supervised domains. Given the nature of the reinforcement learning problem, we argue that standard fine-tuning strategies alone are not enough for efficient transfer in challenging domains. We introduce Behavior Transfer (BT), a technique that leverages pre-trained policies for ex…
National University of Computer and Emerging Science（国立计算机与新兴科学大学）
Deep Reinforcement Learning (DRL) is being used in many domains. One of the biggest advantages of DRL is that it enables the continuous improvement of a learning agent. Secondly, the DRL framework is robust and flexible enough to be applicable to problems of varying nature and domain. Presented work is evidence of using the DRL technique to solve an Optimal Power Flow (OPF) problem. Two classical algorithms have been presented to solve the OPF problem. The drawbacks of the vanilla DRL application are discussed, and an algorithm is suggested to improve the performance. Secondly, a reward functi…
Recent advances in reinforcement learning have proved that given an environment we can learn to perform a task in that environment if we have access to some form of a reward function (dense, sparse or derived from IRL). But most of the algorithms focus on learning a single best policy to perform a given set of tasks. In this paper, we focus on an algorithm that learns to not just perform a task but different ways to perform the same task. As we know when the environment is complex enough there always exists multiple ways to perform a task. We show that using the concept of information maximiza…
How Much Online RL is Enough? Informative Rollouts for Offline Preference Optimization in RLVR
TCS Research Department of CSE（TCS计算机科学系研究部）；IIT Madras（印度理工学院马德拉斯分校）；Department of Data Science & AI（数据科学与人工智能系）；Wadhwani School of Data Science & AI（Wadhwani数据科学与人工智能学院）
Weak-to-Strong Generalization via Direct On-Policy Distillation
SIA-Lab of Tsinghua AIR and ByteDance Seed（清华-字节跳动联合研究中心SIA实验室）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Peking University（北京大学）
Bauman Moscow State Technical University（鲍曼莫斯科国立技术大学）
This paper describes an improvement in Deep Q-learning called Reverse Experience Replay (also RER) that solves the problem of sparse rewards and helps to deal with reward maximizing tasks by sampling transitions successively in reverse order. On tasks with enough experience for training and enough Experience Replay memory capacity, Deep Q-learning Network with Reverse Experience Replay shows competitive results against both Double DQN, with a standard Experience Replay, and vanilla DQN. Also, RER achieves significantly increased results in tasks with a lack of experience and Replay memory capa…
For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it c…
When Reasoning Is Not Enough: Proactive Inquiry for Cross-Modal Safety in Multimodal Large Language Models
Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））；Peking University（北京大学）；School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University（西北工业大学人工智能与光电子学院（iOPEN））
University of Southern California（南加州大学）；Google AI（谷歌人工智能）
Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings. We present an actor-critic algorithm that trains decentralized policies in multi-agent settings, using centrally computed critics that share an attention mechanism which selects relevant information for each agent at every timestep. This attention mechanism enables more effective and scalable learning in complex multi-agent environments, when compared to recent approaches. Our approach is applicable not only to cooperative settings with sh…
University of Oxford（牛津大学）；Google Research（谷歌研究院）；Harvard University（哈佛大学）
We study the problem of planning restless multi-armed bandits (RMABs) with multiple actions. This is a popular model for multi-agent systems with applications like multi-channel communication, monitoring and machine maintenance tasks, and healthcare. Whittle index policies, which are based on Lagrangian relaxations, are widely used in these settings due to their simplicity and near-optimality under certain conditions. In this work, we first show that Whittle index policies can fail in simple and practically relevant RMAB settings, even when the RMABs are indexable. We discuss why the optimalit…
