As more machine learning agents interact with humans, it is increasingly a prospect that an agent trained to perform a task optimally, using only a measure of task performance as feedback, can violate societal norms for acceptable behavior or cause harm. Value alignment is a property of intelligent agents wherein they solely pursue non-harmful behaviors or human-beneficial goals. We introduce an approach to value-aligned reinforcement learning, in which we train an agent with two reward signals: a standard task performance reward, plus a normative behavior reward. The normative behavior reward is derived from a value-aligned prior model previously shown to classify text as normative or non-normative. We show how variations on a policy shaping technique can balance these two sources of reward and produce policies that are both effective and perceived as being more normative. We test our value-alignment technique on three interactive text-based worlds; each world is designed specifically to challenge agents with a task as well as provide opportunities to deviate from the task to engage in normative and/or altruistic behavior.
Reinforcement learning (RL) has shown promise as a tool for engineering safe, ethical, or legal behaviour in autonomous agents. Its use typically relies on assigning punishments to state-action pairs that constitute unsafe or unethical choices. Despite this assignment being a crucial step in this approach, however, there has been limited discussion on generalizing the process of selecting punishments and deciding where to apply them. In this paper, we adopt an approach that leverages an existing framework -- the normative supervisor of (Neufeld et al., 2021) -- during training. This normative …
Georgia Institute of Technology（佐治亚理工学院）；University of Kentucky（肯塔基大学）
Value alignment is a property of an intelligent agent indicating that it can only pursue goals and activities that are beneficial to humans. Traditional approaches to value alignment use imitation learning or preference learning to infer the values of humans by observing their behavior. We introduce a complementary technique in which a value aligned prior is learned from naturally occurring stories which encode societal norms. Training data is sourced from the childrens educational comic strip, Goofus and Gallant. In this work, we train multiple machine learning models to classify natural lang…
Center for Applied Autonomous Sensor Systems（应用自主传感器系统中心）；School of Science and Technology, Örebro University（厄勒布鲁大学科学与技术学院）
Future robots should follow human social norms in order to be useful and accepted in human society. In this paper, we leverage already existing social knowledge in human societies by capturing it in our framework through the notion of social norms. We show how norms can be used to guide a reinforcement learning agent towards achieving normative behavior and apply the same set of norms over different domains. Thus, we are able to: (1) provide a way to intuitively encode social knowledge (through norms); (2) guide learning towards normative behaviors (through an automatic norm reward system); an…
Although the Music Sight Reading process has been studied from the cognitive psychology view points, but the computational learning methods like the Reinforcement Learning have not yet been used to modeling of such processes. In this paper, with regards to essential properties of our specific problem, we consider the value function concept and will indicate that the optimum policy can be obtained by the method we offer without to be getting involved with computing of the complex value functions. Also, we will offer a normative behavioral model for the interaction of the agent with the musical …
GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning
Institute for Interdisciplinary Information Sciences (IIIS), Tsinghua University（交叉信息学院（IIIS）、清华大学）；Language Technologies Institute (LTI), Carnegie Mellon University（语言技术研究所（LTI）、卡内基梅隆大学）
Adaptive reinforcement learning of multi-agent ethically-aligned behaviours: the QSOM and QDSOM algorithms
Univ Lyon（里昂大学）；UCBL（里昂第一大学）；CNRS（法国国家科学研究中心）；INSA Lyon（里昂国立应用科学学院）；Centrale Lyon（里昂中央理工学院）；Univ Lyon 2（里昂第二大学）；LIRIS（里昂信息与图像系统实验室）；Institut Henri Fayol（亨利·法约尔研究所）；MINES Saint-Etienne（圣艾蒂安国立高等矿业学院）；CONFLUENCE: Sciences et Humanités research unit (EA 1598)（汇流：科学与人文研究单元（EA 1598））；Lyon Catholic University（里昂天主教大学）
FAST: A Framework for Aligned Sampling and Training in Parallel Reinforcement Learning for Autonomous Driving
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与运载学院）；Didi Voyager Labs, DiDi Autonomous Driving（滴滴自动驾驶沃芽实验室）；College of AI, Tsinghua University（清华大学人工智能学院）
Optimal decision making with limited or no information in stochastic environments where multiple agents interact is a challenging topic in the realm of artificial intelligence. Reinforcement learning (RL) is a popular approach for arriving at optimal strategies by predicating stimuli, such as the reward for following a strategy, on experience. RL is heavily explored in the single-agent context, but is a nascent concept in multiagent problems. To this end, I propose several principled model-free and partially model-based reinforcement learning approaches for several multiagent settings. In the …
Active Advantage-Aligned Online Reinforcement Learning with Offline Data
Department of Computer Science, University of Chicago（计算机科学系，芝加哥大学）；Department of Statistics and Data Science, Yale University（统计与数据科学系，耶鲁大学）；Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
CrystalGRPO: Target-Aligned and Coverage-Preserving Reinforcement Learning for Flow-Based Crystal Structure Prediction
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；North Carolina Battery Complexity, Autonomous Vehicle and Electrification (BATT CAVE) Research Center（北卡罗来纳州电池复杂性、自动驾驶与电气化（BATT CAVE）研究中心）
Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation
Shanghai Jiao Tong University（上海交通大学）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；BIGAI（北京智源人工智能研究院）；Tencent Robotics X（腾讯Robotics X实验室）；University of Alberta（阿尔伯塔大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards
Tsinghua University（清华大学）；Renmin University of China（中国人民大学）；Institute of Information Engineering, CAS（中国科学院信息工程研究所）；USTC（中国科学技术大学）；The Australian National University（澳大利亚国立大学）；Peking University（北京大学）；University of Macau（澳门大学）
Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learning
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）；East China Normal University（华东师范大学）；Tongji University（同济大学）
STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning
Beijing Key Laboratory of Embodied Intelligence Systems, Department of Automation, Tsinghua University, Beijing, China（北京智能体智能系统重点实验室，自动化系，清华大学，北京，中国）；The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China（复杂系统认知与决策智能重点实验室，自动化研究所，中国科学院，北京，中国）
University of Toronto（多伦多大学）；Vector Institute（Vector研究所）
Reinforcement learning (RL) agents have traditionally been tasked with maximizing the value function of a Markov decision process (MDP), either in continuous settings, with fixed discount factor $γ< 1$, or in episodic settings, with $γ= 1$. While this has proven effective for specific tasks with well-defined objectives (e.g., games), it has never been established that fixed discounting is suitable for general purpose use (e.g., as a model of human preferences). This paper characterizes rationality in sequential decision making using a set of seven axioms and arrives at a form of discounting th…
FARCA: Fact-Aligned Reliability-Aware Credit Assignment for Reinforcement Learning with Factual Supervision
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
E-MRL: Cross-view Aligned Evidence-driven Multimodal Reinforcement Learning for Reliable 3D Tumor Analysis
Zhejiang University（浙江大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Hupan Lab（华平实验室）；Huazhong University of Science and Technology（华中科技大学）；East China Normal University（华东师范大学）；Shanghai Jiao Tong University（上海交通大学）
Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filtering
City University of Hong Kong；University of Illinois Urbana-Champaign；Tencent；Institute of Artificial Intelligence, China Telecom (TeleAI)；Tsinghua University. Corresponding Author
Mechanical Conscience: A Mathematical Framework for Dependability of Machine Intelligence
Department of Information Security Engineering, Soonchunhyang University（信息安全工程系，Soonchunhyang大学）；Department of Mobility Convergence Security, Soonchunhyang University（移动融合安全系，Soonchunhyang大学）
Multi-objective Reinforcement Learning: A Tool for Pluralistic Alignment
Institute of Innovation, Science and Sustainability（创新、科学与可持续发展研究所）；Federation University Australia（澳大利亚联邦大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；School of IT（信息技术学院）；Deakin University（迪肯大学）
Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)
Sch. of CS, Sch. of AIS, Sch. of AI, Shanghai Jiao Tong University（计算机科学学院、人工智能科学学院、人工智能学院，上海交通大学）；Institute of Trustworthy Embodied AI, Fudan University（可信具身人工智能研究院，复旦大学）；AgiBot
University of Bristol（布里斯托大学）；CortAIx, Thales（泰雷兹公司 CortAIx）
The potential of reinforcement learning (RL) to deliver aligned and performant agents is partially bottlenecked by the reward engineering problem. One alternative to heuristic trial-and-error is preference-based RL (PbRL), where a reward function is inferred from sparse human feedback. However, prior PbRL methods lack interpretability of the learned reward structure, which hampers the ability to assess robustness and alignment. We propose an online, active preference learning algorithm that constructs reward functions with the intrinsically interpretable, compositional structure of a tree. Usi…
Quantile-Coupled Flow Matching for Distributional Reinforcement Learning
Oxford Robotics Institute, University of Oxford（牛津大学机器人研究所）；Bristol Robotics Laboratory, University of the West of England（布里斯托尔机器人实验室，西英格兰大学）；Dyson Institute of Engineering & Technology（戴森工程与技术研究院）
