DiAReL: Reinforcement Learning with Disturbance Awareness for Robust Sim2Real Policy Transfer in Robot Control
Department of Computer Engineering, School of Computation, Information and Technology, Technical University of Munich(计算机工程系,计算、信息与技术学院,慕尼黑技术大学);Rosenheim University of Applied Sciences(罗森海姆应用技术大学)
Mohammadhossein Malmir, Josip Josifovski, Noah Klarmann, Alois Knoll
备注:
Accepted for publication in IEEE Transactions on Control Systems Technology (TCST)
Delayed Markov decision processes (DMDPs) fulfill the Markov property by augmenting the state space of agents with a finite time window of recently committed actions. In reliance on these state augmentations, delay-resolved reinforcement learning algorithms train policies to learn optimal interactions with environments featuring observation or action delays. Although such methods can be directly trained on the real robots, due to sample inefficiency, limited resources, or safety constraints, a common approach is to transfer models trained in simulation to the physical robot. However, robotic simulations rely on approximated models of the physical systems, which hinders the sim2real transfer. In this work, we consider various uncertainties in modeling the robot or environment dynamics as unknown intrinsic disturbances applied to the system input. We introduce the disturbance-augmented Markov decision process (DAMDP) in delayed settings as a novel representation to incorporate disturbance estimation in training on-policy reinforcement learning algorithms. The proposed method is validated across several metrics on learning robotic reaching and pushing tasks and compared with disturbance-unaware baselines. The results show that the disturbance-augmented models can achieve higher stabilization and robustness in the control response, which in turn improves the prospects of successful sim2real transfer.
RL-based Control of UAS Subject to Significant Disturbance
Smart Mechatronics and Robotics research group, Saxion University of Applied Science(智能机电与机器人研究组,萨克斯Ion应用科学大学);Robotics and Mechatronics research group, Faculty of Electrical Engineering, Mathematics & Computer Science, University of Twente(机器人与机电研究组,电子工程、数学与计算机科学学院,特文特大学)
Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimization
CAS Key Laboratory of Technology in GIPAS, University of Science and Technology of China(中国科学院空间信息处理与应用系统技术重点实验室,中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
Opponent Learning Awareness and Modelling in Multi-Objective Normal Form Games
Vrije Universiteit Brussel(布鲁塞尔自由大学);Universiteit van Amsterdam(阿姆斯特丹大学);National University of Ireland Galway(爱尔兰国立大学戈尔韦分校);HU Univ. of Appl. Sci. Utrecht(乌得勒支HU应用科学大学)
Proximal Learning With Opponent-Learning Awareness
University of Toronto(多伦多大学);Vector Institute(向量研究所);University of Oxford(牛津大学);FLAIR(未来人工智能研究院)
具有对手学习意识的学习(LOLA)(Foerster et al. [2018 a])是一种多智能体强化学习算法,通常在部分竞争环境中学习基于互惠的合作。然而,LOLA往往无法学习更复杂的政策空间参数化的神经网络,部分原因是更新规则是敏感的政策参数化的行为。这个问题在对手建模设置中尤其突出,其中对手的策略是未知的,并且必须从观察中推断;在这种设置中,LOLA是指定不良的,因为行为上等同的对手策略可能导致不等同的更新。为了解决这一缺点,我们重新解释LOLA近似近端运营商,然后推导出一个新的算法,近端LOLA(POLA),它直接使用的近端配方。与LOLA不同,POLA更新是参数化不变的,在这个意义上说,当最近的目标有一个唯一的最佳,行为等效的政策导致行为等效的更新。然后,我们提出了实际的近似理想的POLA更新,我们评估在几个部分竞争的环境中的功能近似和对手建模。这从经验上证明,POLA比LOLA更可靠地实现基于互惠的合作。
MAD: A Magnitude And Direction Policy Parametrization for Stability Constrained Reinforcement Learning
Institute of Mechanical Engineering, École Polytechnique Fédérale de Lausanne (EPFL)(瑞士联邦理工学院机械工程学院);Department of Engineering Science, University of Oxford(牛津大学工程科学系);Institute for Data Science Foundations, Hamburg University of Technology (TUHH)(汉堡技术大学数据科学基础研究所);School of Electrical Engineering and Computer Science and Digital Futures, KTH Royal Institute of Technology(皇家理工学院电气工程与计算机科学学院及数字未来学院)
VidAct: Learning Manipulation from In-the-Wild Videos with Object-Centric 3D Awareness
Technical University of Munich(慕尼黑工业大学);Peking University(北京大学);Huawei Heisenberg Research Center(华为海森堡研究中心);Huawei CloudRobo Lab(华为云机器人实验室);Nanjing University(南京大学)
CARO: Contact-Agnostic Residual Observation for Zero-Shot Robust Quadruped Locomotion
School of Aeronautic Science and Engineering, Beihang University(北京航空航天大学航空科学与工程学院);School of Automation Science and Electrical Engineering, Beihang University(北京航空航天大学自动化科学与电气工程学院)
Dual Self-Awareness Value Decomposition Framework without Individual Global Max for Cooperative Multi-Agent Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
针对合作多智能体强化学习中IGM原则限制问题,提出双自感知价值分解框架,结合自我策略与 alter ego 价值函数并引入反自我探索机制,实现完全无需IGM的价值分解并在多任务中表现优异。
Disturbance-Aware Adaptive Compensation in Hybrid Force-Position Locomotion Policy for Legged Robots
Department of Automation, Shanghai Jiao Tong University(自动化系,上海交通大学);Department of Computer Science, Shanghai Jiao Tong University(计算机科学系,上海交通大学);MoE Key Lab of Artificial Intelligence and AI Institute, Shanghai Jiao Tong University(人工智能与AI研究院,上海交通大学)
Simulation-Based Evaluation of Residual DDPG-Augmented Computed Torque Control for a Cable-Driven Lower-Limb Rehabilitation Robot Model under Disturbances and Parametric Uncertainties
SAC-Loco: Safe and Adjustable Compliant Quadrupedal Locomotion
Department of Electrical and Computer Engineering, National University of Singapore(新加坡国立大学电子与计算机工程系);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院)
DoublyAware: Dual Planning and Policy Awareness for Temporal Difference Learning in Humanoid Locomotion
University of Texas at Arlington(德克萨斯大学阿灵顿分校);Intelligent Autonomous Systems Lab(智能自主系统实验室);German Research Center for AI (DFKI)(德国人工智能研究中心);Automation & Control Institute (ACIN)(自动化与控制研究所);Austrian Institute of Technology (AIT) GmbH(奥地利技术研究所)
AM-Bench: A Modular Simulation Suite and Benchmark for Aerial Manipulation Policy Learning
The Robotics Institute, School of Computer Science, Carnegie Mellon University(卡内基梅隆大学计算机学院机器人研究所);Department of Aerospace Engineering, Pennsylvania State University(宾夕法尼亚州立大学航空航天工程系);School of Mechanical Engineering, Kyung Hee University(庆熙大学机械工程学院)