Specifying rewards for reinforcement learned (RL) agents is challenging. Preference-based RL (PbRL) mitigates these challenges by inferring a reward from feedback over sets of trajectories. However, the effectiveness of PbRL is limited by the amount of feedback needed to reliably recover the structure of the target reward. We present the PRIor Over Rewards (PRIOR) framework, which incorporates priors about the structure of the reward function and the preference feedback into the reward learning process. Imposing these priors as soft constraints on the reward learning objective reduces the amount of feedback required by half and improves overall reward recovery. Additionally, we demonstrate that using an abstract state space for the computation of the priors further improves the reward learning and the agent's performance.
HISR: Hindsight Information Modulated Segmental Process Rewards For Multi-turn Agentic Reinforcement Learning
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航空航天信息研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent Hunyuan（腾讯文脉）；School of Computer Science and Technology, Tianjin University（天津大学计算机科学与技术学院）
Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following
Zhejiang University（浙江大学）；Nanyang Technological University（南洋理工大学）；Dalian University of Technology（大连理工大学）；University of Science and Technology of China（中国科学技术大学）；Alibaba Cloud Computing（阿里云计算）；Chinese Academy of Sciences（中国科学院）
HiR 提出了一种样本高效强化学习框架，通过将失败尝试视为成功来提升指令跟随任务的性能，利用双偏好学习实现高效优化。
We define a novel neuro-symbolic framework, argumentative reward learning, which combines preference-based argumentation with existing approaches to reinforcement learning from human feedback. Our method improves prior work by generalising human preferences, reducing the burden on the user and increasing the robustness of the reward model. We demonstrate this with a number of experiments.
Reinforcement learning is an appropriate and successful method to robustly perform low-level robot control under noisy conditions. Symbolic action planning is useful to resolve causal dependencies and to break a causally complex problem down into a sequence of simpler high-level actions. A problem with the integration of both approaches is that action planning is based on discrete high-level action- and state spaces, whereas reinforcement learning is usually driven by a continuous reward function. However, recent advances in reinforcement learning, specifically, universal value function approx…
Controllability in preference-conditioned multi-objective reinforcement learning
LTCI, Télécom Paris, Institut Polytechnique de Paris（LTCI，巴黎电信学院，巴黎理工学院）；University of California, Berkeley（加州大学伯克利分校）；TU Delft（代尔夫特理工大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Generating complex behaviors that satisfy the preferences of non-expert users is a crucial requirement for AI agents. Interactive reward learning from trajectory comparisons (a.k.a. RLHF) is one way to allow non-expert users to convey complex objectives by expressing preferences over short clips of agent behaviors. Even though this parametric method can encode complex tacit knowledge present in the underlying tasks, it implicitly assumes that the human is unable to provide richer feedback than binary preference labels, leading to intolerably high feedback complexity and poor user experience. W…
We study an online linear classification problem, in which the data is generated by strategic agents who manipulate their features in an effort to change the classification outcome. In rounds, the learner deploys a classifier, and an adversarially chosen agent arrives, possibly manipulating her features to optimally respond to the learner. The learner has no knowledge of the agents' utility functions or "real" features, which may vary widely across agents. Instead, the learner is only able to observe their "revealed preferences" --- i.e. the actual manipulated feature vectors they provide. For…
Humans have the ability to rapidly understand rich combinatorial concepts from limited data. Here we investigate this ability in the context of auditory signals, which have been evolved in a cultural transmission experiment to study the emergence of combinatorial structure in language. We propose a neuro-symbolic generative model which combines the strengths of previous approaches to concept learning. Our model performs fast inference drawing on neural network methods, while still retaining the interpretability and generalization from limited data seen in structured generative approaches. This…
Preference Elicitation for Policy Optimization and Application to Aligning Heart Transplantation with Human Values
Carnegie Mellon University（卡内基梅隆大学）；Medical University of South Carolina（南卡罗来纳医科大学）；Strategy Robot, Inc.（策略机器人公司）；Strategic Machine, Inc.（战略机器公司）；Optimized Markets, Inc.（优化市场公司）
