Recently, there has been an increasing need to develop agents capable of solving multiple tasks within the same environment, especially when these tasks are naturally associated with language. In this work, we propose a novel approach that leverages combinations of pre-trained (language, policy) pairs to establish an efficient transfer pipeline. Our algorithm is inspired by the principles of Contrastive Language-Image Pretraining (CLIP) in Computer Vision, which aligns representations across different modalities under the philosophy that ''two modalities representing the same concept should have similar representations.'' The central idea here is that the instruction and corresponding policy of a task represent the same concept, the task itself, in two different modalities. Therefore, by extending the idea of CLIP to RL, our method creates a unified representation space for natural language and policy embeddings. Experimental results demonstrate the utility of our algorithm in achieving faster transfer across tasks.
Intelligent Collaborative Optimization for Rubber Tyre Film Production Based on Multi-path Differentiated Clipping Proximal Policy Optimization
State Key Laboratory of Physical Oceanography and the Faculty of Information Science and Engineering, Ocean University of China(物理海洋学国家重点实验室和信息科学与工程学院,中国海洋大学);School of Mathematical and Computer Sciences, Heriot-Watt University(数学与计算机科学学院,赫瑞-沃德大学);Mesnac Co., Ltd.(梅斯纳克公司)
Dimension-Wise Importance Sampling Weight Clipping for Sample-Efficient Reinforcement Learning
KAIST(韩国科学技术院)
在诸如近端策略优化(PPO)的基于重要性采样(IS)的强化学习算法中,IS权重通常被裁剪以避免学习中的大方差。然而,从裁剪统计的政策更新引起大的偏见,在任务中具有高的动作维度,和偏见裁剪使得它很难重用旧的样本与大的IS权重。在本文中,我们认为PPO,一个代表性的政策算法,并提出了其改进的维度上的IS权重裁剪,分别剪辑每个动作维度的IS权重,以避免大的偏见,并自适应地控制IS权重绑定从当前策略的策略更新。这种新技术能够有效地学习高动作维度的任务,并重复使用旧的样本,如在离策略学习,以提高样本效率。数值实验结果表明,该算法在各种Open AI Gym任务中的性能优于PPO和其他RL算法。
CIM-PPO:Proximal Policy Optimization with Liu-Correntropy Induced Metric
College of Science, National University of Defense Technology(国防科技大学理学院);School of System Science and Engineering, Sun-Yat-Sen University(中山大学系统科学与工程学院);College of System Engineering, National University of Defense Technology(国防科技大学系统工程学院)
Department of Foundation Model, 2012 Labs, Huawei(华为基础模型部门,2012实验室);Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University(上海智能自主系统研究院,同济大学);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);College of Intelligence and Computing, Tianjin University(天津大学智能与计算学院)
PRPO: Aligning Process Reward with Outcome Reward in Policy Optimization
Shanghai University(上海大学);University of Science and Technology of China(中国科学技术大学);Shanghai Jiaotong University(上海交通大学);Artificial Intelligence Incubation and Innovation Institute, Fudan University(复旦大学人工智能孵化与创新研究院);Shanghai Academy of AI for Science(上海人工智能科学研究院)
Preference Elicitation for Policy Optimization and Application to Aligning Heart Transplantation with Human Values
Carnegie Mellon University(卡内基梅隆大学);Medical University of South Carolina(南卡罗来纳医科大学);Strategy Robot, Inc.(策略机器人公司);Strategic Machine, Inc.(战略机器公司);Optimized Markets, Inc.(优化市场公司)
Harness Policy Distillation: Aligning Agent Policies with Runtime Semantics
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院)