Adversarial optimization algorithms that explicitly search for flaws in agents' policies have been successfully applied to finding robust and diverse policies in multi-agent settings. However, the success of adversarial optimization has been largely limited to zero-sum settings because its naive application in cooperative settings leads to a critical failure mode: agents are irrationally incentivized to self-sabotage, blocking the completion of tasks and halting further learning. To address this, we introduce Rationality-preserving Policy Optimization (RPO), a formalism for adversarial optimization that avoids self-sabotage by ensuring agents remain rational--that is, their policies are optimal with respect to some possible partner policy. To solve RPO, we develop Rational Policy Gradient (RPG), which trains agents to maximize their own reward in a modified version of the original game in which we use opponent shaping techniques to optimize the adversarial objective. RPG enables us to extend a variety of existing adversarial optimization algorithms that, no longer subject to the limitations of self-sabotage, can find adversarial examples, improve robustness and adaptability, and learn diverse policies. We empirically validate that our approach achieves strong performance in several popular cooperative and general-sum environments. Our project page can be found at https://rational-policy-gradient.github.io.
Learnable Game-theoretic Policy Optimization for Data-centric Self-explanation Rationalization
School of Computer and Information Technology, Shanxi University(山西大学计算机与信息学院);School of Computer and Information Systems, Singapore Management University(新加坡管理学院计算机与信息学院);Institute for Infocomm Research, A*Star(A*Star信息与通信研究所)
Artificial intelligence, rationalization, and the limits of control in the public sector: the case of tax policy optimization
Oxford Internet Institute, University of Oxford(牛津大学牛津互联网研究院);Center for Information Technology Policy, Princeton University(普林斯顿大学信息技术政策中心);Center for Advanced Study in the Social Sciences, Stanford University(斯坦福大学社会科学高等研究中心)
GRACE: Generative Representation Learning via Contrastive Policy Optimization
University of Illinois Urbana–Champaign(伊利诺伊大学厄巴纳-香槟分校);Australian National University(澳大利亚国立大学);Hong Kong University of Science and Technology(香港科学与技术大学);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);University of Washington(华盛顿大学)
Beyond Mode Collapse: Distribution Matching for Diverse Reasoning
Tongji University(同济大学);Independent(独立);Shanghai AI Laboratory(上海人工智能实验室);Zhejiang University(浙江大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
RePolicy: Learning to Select and Apply Safety Policies for Agent Safeguards
Zhejiang University(浙江大学);Zhongguancun Academy(中关村学院);University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学)
TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment
Huazhong University of Science and Technology(华中科技大学);Kuaishou Technology(快手科技);Nanyang Technological University(南洋理工大学);Wuhan University(武汉大学);Tsinghua University(清华大学)
More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Exploration
Tongji University(同济大学);Hong Kong Polytechnic University(香港理工大学);Shanghai AI Lab(上海人工智能实验室);National University of Singapore(新加坡国立大学);University of Electronic Science and Technology of China(电子科技大学)
奖励函数是强化学习(RL)中必不可少的,作为激励智能体解决给定任务的指导信号,然而,也是众所周知的难以设计。在许多情况下,只有不完美的奖励是可用的,这对RL代理造成了重大的性能损失。在这项研究中,我们提出了一个统一的离线策略优化方法,\textit{RGM (Reward Gap Minimization)},它可以巧妙地处理不同类型的不完美奖励。RGM被制定为一个双层优化问题:上层优化的奖励校正项,执行访问分布匹配w.r.t.一些专家数据;较低层用校正的奖励解决悲观RL问题。通过利用较低层的二元性,我们得到了一个易于处理的算法,使基于采样的学习没有任何在线交互。综合实验表明,RGM实现了优越的性能,现有的方法在不同的设置不完美的奖励。此外,RGM可以有效地纠正错误的或不一致的奖励对专家的偏好和检索有用的信息,从有偏见的奖励。