It is often difficult to hand-specify what the correct reward function is for a task, so researchers have instead aimed to learn reward functions from human behavior or feedback. The types of behavior interpreted as evidence of the reward function have expanded greatly in recent years. We've gone from demonstrations, to comparisons, to reading into the information leaked when the human is pushing the robot away or turning it off. And surely, there is more to come. How will a robot make sense of all these diverse types of behavior? Our key insight is that different types of behavior can be interpreted in a single unifying formalism - as a reward-rational choice that the human is making, often implicitly. The formalism offers both a unifying lens with which to view past work, as well as a recipe for interpreting new sources of information that are yet to be uncovered. We provide two examples to showcase this: interpreting a new feedback type, and reading into how the choice of feedback itself leaks information about the reward.
National University of Singapore（新加坡国立大学）；Massachusetts Institute of Technology（麻省理工学院）
This paper presents a novel nonmyopic adaptive Gaussian process planning (GPP) framework endowed with a general class of Lipschitz continuous reward functions that can unify some active learning/sensing and Bayesian optimization criteria and offer practitioners some flexibility to specify their desired choices for defining new tasks/problems. In particular, it utilizes a principled Bayesian sequential decision problem framework for jointly and naturally optimizing the exploration-exploitation trade-off. In general, the resulting induced GPP policy cannot be derived exactly due to an uncountabl…
German Research Center for AI (DFKI)（德国人工智能研究中心（DFKI））；Centre for Cognitive Science（认知科学中心）
Recent methods for imitation learning directly learn a $Q$-function using an implicit reward formulation rather than an explicit reward function. However, these methods generally require implicit reward regularization to improve stability and often mistreat absorbing states. Previous works show that a squared norm regularization on the implicit reward function is effective, but do not provide a theoretical analysis of the resulting properties of the algorithms. In this work, we show that using this regularizer under a mixture distribution of the policy and the expert provides a particularly il…
University of California, San Diego（加州大学圣地亚哥分校）；Victoria University（维多利亚大学）
In reinforcement learning, a decision needs to be made at some point as to whether it is worthwhile to carry on with the learning process or to terminate it. In many such situations, stochastic elements are often present which govern the occurrence of rewards, with the sequential occurrences of positive rewards randomly interleaved with negative rewards. For most practical learners, the learning is considered useful if the number of positive rewards always exceeds the negative ones. A situation that often calls for learning termination is when the number of negative rewards exceeds the number …
Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill
Qwen Large Model Application Team, Alibaba（通义千问大模型应用团队，阿里巴巴）；Sun Yat-sen University（中山大学）；The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）；ETH Zürich University of Zurich（苏黎世联邦理工学院）
We study the implicit bias of gradient flow (i.e., gradient descent with infinitesimal step size) on linear neural network training. We propose a tensor formulation of neural networks that includes fully-connected, diagonal, and convolutional networks as special cases, and investigate the linear version of the formulation called linear tensor networks. With this formulation, we can characterize the convergence direction of the network parameters as singular vectors of a tensor defined by the network. For $L$-layer linear tensor networks that are orthogonally decomposable, we show that gradient…
FAIR Meta AI（元人工智能基础人工智能研究院）；University of Pennsylvania（宾夕法尼亚大学）
Reward and representation learning are two long-standing challenges for learning an expanding set of robot manipulation skills from sensory observations. Given the inherent cost and scarcity of in-domain, task-specific robot data, learning from large, diverse, offline human videos has emerged as a promising path towards acquiring a generally useful visual representation for control; however, how these human videos can be used for general-purpose reward learning remains an open question. We introduce $\textbf{V}$alue-$\textbf{I}$mplicit $\textbf{P}$re-training (VIP), a self-supervised pre-train…
TU Dortmund University（多特蒙德工业大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；TU Darmstadt（达姆施塔特工业大学）
Non-linear kernel methods can be approximated by fast linear ones using suitable explicit feature maps allowing their application to large scale problems. We investigate how convolution kernels for structured data are composed from base kernels and construct corresponding feature maps. On this basis we propose exact and approximative feature maps for widely used graph kernels based on the kernel trick. We analyze for which kernels and graph properties computation by explicit feature maps is feasible and actually more efficient. In particular, we derive approximative, explicit feature maps for …
Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学计算机科学与工程保罗·G·艾伦学校）；Max Planck Institute for Intelligent Systems, Tübingen, Germany（德国图宾根马克斯·普朗克智能系统研究所）；Jinesis Lab, University of Toronto & Vector Institute（多伦多大学Jinesis实验室及向量研究所）
AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
Rutgers University（新泽西州立大学）；Nanyang Technological University（南洋理工大学）；University of Connecticut（康涅狄格大学）；Fudan University（复旦大学）；NVIDIA Research（NVIDIA研究）；Red Hat AI Innovation（红帽AI创新）；MIT-IBM Watson AI Lab（MIT-IBM沃森AI实验室）；Massachusetts Institute of Technology（麻省理工学院）
Politecnico di Milano（米兰理工大学）；Bocconi University（博科尼大学）
We study online learning problems in which a decision maker has to take a sequence of decisions subject to $m$ long-term constraints. The goal of the decision maker is to maximize their total reward, while at the same time achieving small cumulative constraints violation across the $T$ rounds. We present the first best-of-both-world type algorithm for this general class of problems, with no-regret guarantees both in the case in which rewards and constraints are selected according to an unknown stochastic model, and in the case in which they are selected at each round by an adversary. Our algor…
Shanghai AI Lab（上海人工智能实验室）；Tsinghua University（清华大学）；University of Illinois Urbana-Champaign（伊利诺伊大学香槟分校）；Peking University（北京大学）；Shanghai Jiaotong University（上海交通大学）；CUHK（香港中文大学）
University of California, San Diego（加州大学圣地亚哥分校）；University of Southampton（南安普顿大学）
This paper investigates the adversarial Bandits with Knapsack (BwK) online learning problem, where a player repeatedly chooses to perform an action, pays the corresponding cost, and receives a reward associated with the action. The player is constrained by the maximum budget $B$ that can be spent to perform actions, and the rewards and the costs of the actions are assigned by an adversary. This problem has only been studied in the restricted setting where the reward of an action is greater than the cost of the action, while we provide a solution in the general setting. Namely, we propose EXP3.…
Utility-Based Reinforcement Learning: Unifying Single-objective and Multi-objective Reinforcement Learning
Federation University Australia（澳大利亚联邦大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；University of Galway（高威大学）；Deakin University（迪肯大学）；Linköping University（林雪平大学）；Universidade do Vale do Rio dos Sinos（南里约 dos Sinos大学）；Vrije Universiteit Brussel（布鲁塞尔自由大学）
PREFINE: Preference-Based Implicit Reward and Cost Fine-Tuning for Safety Alignment
TCS Research, \ of CSE, IIT Madras India；Department of Computing Science, \ of Alberta Canada；Qatar Computing Research Institute, \ Bin Khalifa University Qatar；Department of Data Science \& AI, Wadhwani School of Data Science \& AI, IIT Madras India；TCS Research, \ of CSE, IIT Madras；Department of Computing Science, \ of Alberta；Qatar Computing Research Institute, \ Bin Khalifa University；Department of Data Science \& AI, Wadhwani School of Data Science \& AI, IIT Madras
Oregon State University（俄勒冈州立大学）；Horizon Robotics（地平线机器人）
Efficient exploration remains a challenging problem in reinforcement learning, especially for those tasks where rewards from environments are sparse. A commonly used approach for exploring such environments is to introduce some "intrinsic" reward. In this work, we focus on model uncertainty estimation as an intrinsic reward for efficient exploration. In particular, we introduce an implicit generative modeling approach to estimate a Bayesian uncertainty of the agent's belief of the environment dynamics. Each random draw from our generative model is a neural network that instantiates the dynamic…
Learning to Reason About Implicit Multimodal Harm: From Diagnosis to Generalization
Max Planck Institute for Informatics（马克斯·普朗克院信息研究所）；Saarland Informatics Campus（萨尔兰州信息校园）；Saarland University（萨尔兰州大学）；The University of Edinburgh（爱丁堡大学）；Samsung AI Center, Cambridge（三星AI中心，剑桥）
DavIR: Data Selection via Implicit Reward for Large Language Models
ByteDance, Inc.（字节跳动公司）；Cold Spring Harbor Laboratory（冷泉港实验室）；Shanghai Jiao Tong University（上海交通大学）；National University of Singapore（新加坡国立大学）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Although reinforcement learning has found widespread use in dense reward settings, training autonomous agents with sparse rewards remains challenging. To address this difficulty, prior work has shown promising results when using not only task-specific demonstrations but also task-agnostic albeit somewhat related demonstrations. In most cases, the available demonstrations are distilled into an implicit prior, commonly represented via a single deep net. Explicit priors in the form of a database that can be queried have also been shown to lead to encouraging results. To better benefit from availa…
SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Baichuan Inc.（百川智能）
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University（宾夕法尼亚州立大学人工智能研究实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent AI Lab（腾讯人工智能实验室）
Stony Brook University（石溪大学）；Microsoft Research NYC（微软研究院纽约分院）；Texas A&M University（德州农工大学）；University of Virginia（弗吉尼亚大学）
In an era of countless content offerings, recommender systems alleviate information overload by providing users with personalized content suggestions. Due to the scarcity of explicit user feedback, modern recommender systems typically optimize for the same fixed combination of implicit feedback signals across all users. However, this approach disregards a growing body of work highlighting that (i) implicit signals can be used by users in diverse ways, signaling anything from satisfaction to active dislike, and (ii) different users communicate preferences in different ways. We propose applying …
For adversarial imitation learning algorithms (AILs), no true rewards are obtained from the environment for learning the strategy. However, the pseudo rewards based on the output of the discriminator are still required. Given the implicit reward bias problem in AILs, we design several representative reward function shapes and compare their performances by large-scale experiments. To ensure our results' reliability, we conduct the experiments on a series of Mujoco and Box2D continuous control tasks based on four different AILs. Besides, we also compare the performance of various reward function…
Emotions are intimately tied to motivation and the adaptation of behavior, and many animal species show evidence of emotions in their behavior. Therefore, emotions must be related to powerful mechanisms that aid survival, and, emotions must be evolutionary continuous phenomena. How and why did emotions evolve in nature, how do events get emotionally appraised, how do emotions relate to cognitive complexity, and, how do they impact behavior and learning? In this article I propose that all emotions are manifestations of reward processing, in particular Temporal Difference (TD) error assessment. …
