Department of Computer Science and Technology, University of Cambridge, United Kingdom（计算机科学与技术系，剑桥大学，英国）；Robotics and Biology Laboratory, Technische Universität Berlin（机器人与生物学实验室，柏林技术大学）；Science of Intelligence (SCIoI), Cluster of Excellence, Berlin, Germany（智能科学（SCIoI），卓越中心，柏林，德国）；Robotics Institute Germany（德国机器人研究所）
Eduardo Sebastián, Adrian Pfisterer, Vito Mengers, Oliver Brock, Amanda Prorok
Robot learning must produce policies that generalize to new combinations of constraints, teammates, and environments. To achieve this, we must structurally factor the policy, which is a choice that dictates what generalizes, what requires retraining, and what remains entangled. Existing methods span a wide spectrum, from expecting structure to emerge from data scaling, to hand-designing it via hierarchies, skill libraries or learned specializations. In this paper, we study what we argue is the most fundamental factorization in robotics: separating the world from the task. We investigate the conditions under which this factorization is principled. World factors are properties of the embodied system and the environment; they exist independently of intent. Task factors are defined by the task's logic over what the world admits. We formalize this asymmetry through Bayesian model evidence: it aligns with the data-generating process, maintains high likelihood through an analytical world model, and reduces the Occam razor's penalty on task parameters. We instantiate this factorization by pairing AICON, a differentiable graph of recursive estimators and interconnections that is compositional, operates without task-specific data, and propagates cost gradients to actuators, with a compact, learned policy that modulates gradient paths. Gradients serve as the interface between the two factors: they carry world structure through the graph and task structure through costs, enabling low-dimensional learning while preserving structural generalization. We test the world/task factorization across three problems that encompass heterogeneous robots, environments, task logic and sensorimotor modalities. Our framework outperforms end-to-end baselines and analytical heuristics in all settings, generalizes zero-shot to out-of-distribution configurations, and transfers to real hardware without retraining.
To solve tasks in complex environments, robots need to learn from experience. Deep reinforcement learning is a common approach to robot learning but requires a large amount of trial and error to learn, limiting its deployment in the physical world. As a consequence, many advances in robot learning rely on simulators. On the other hand, learning inside of simulators fails to capture the complexity of the real world, is prone to simulator inaccuracies, and the resulting behaviors do not adapt to changes in the world. The Dreamer algorithm has recently shown great promise for learning from small …
In this work, we explore self-supervised visual pre-training on images from diverse, in-the-wild videos for real-world robotic tasks. Like prior work, our visual representations are pre-trained via a masked autoencoder (MAE), frozen, and then passed into a learnable control module. Unlike prior work, we show that the pre-trained representations are effective across a range of real-world robotic tasks and embodiments. We find that our encoder consistently outperforms CLIP (up to 75%), supervised ImageNet pre-training (up to 81%), and training from scratch (up to 81%). Finally, we train a 307M p…
Multi-robot systems can benefit from reinforcement learning (RL) algorithms that learn behaviours in a small number of trials, a property known as sample efficiency. This research thus investigates the use of learned world models to improve sample efficiency. We present a novel multi-agent model-based RL algorithm: Multi-Agent Model-Based Policy Optimization (MAMBPO), utilizing the Centralized Learning for Decentralized Execution (CLDE) framework. CLDE algorithms allow a group of agents to act in a fully decentralized manner after training. This is a desirable property for many systems compris…
School of Informatics, Computing, and Engineering（信息学、计算与工程学院）；Indiana University Bloomington（印第安纳大学伯明顿分校）
Learning to control robots directly based on images is a primary challenge in robotics. However, many existing reinforcement learning approaches require iteratively obtaining millions of robot samples to learn a policy, which can take significant time. In this paper, we focus on learning a realistic world model capturing the dynamics of scene changes conditioned on robot actions. Our dreaming model can emulate samples equivalent to a sequence of images from the actual environment, technically by learning an action-conditioned future representation/scene regressor. This allows the agent to lear…
World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation
The State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Zhongguancun Academy（中关村学院）；Beijing Zhongke Huiling Robot Technology Co（北京中科创联机器人技术有限公司）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）
IGen: Scalable Data Generation for Robot Learning from Open-World Images
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；The University of Hong Kong（香港大学）；Beijing University of Chemical Technology（北京化工大学）；Shanghai Jiao Tong University（上海交通大学）；Shenzhen University of Infomation Technology（深圳信息大学）
Reinforcement learning has been demonstrated as a flexible and effective approach for learning a range of continuous control tasks, such as those used by robots to manipulate objects in their environment. But in robotics particularly, real-world rollouts are costly, and sample efficiency can be a major limiting factor when learning a new skill. In game environments, the use of world models has been shown to improve sample efficiency while still achieving good performance, especially when images or other rich observations are provided. In this project, we explore the use of a world model in a d…
WorldSample: Closed-loop Real-robot RL with World Modelling
PINE Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（南洋理工大学电气与电子工程学院PINE实验室）；Department of Electronic Engineering, Tsinghua University, Beijing, China（清华大学电子工程系）；School of Automation, Central South University, Changsha, China（中南大学自动化学院）；School of Automation, Beijing University of Posts and Telecommunications, Beijing, China（北京邮电大学自动化学院）
World Model for Robot Learning: A Comprehensive Survey
Nanyang Technological University（南洋理工大学）；University of California, Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）；The University of Tokyo（东京大学）；University of Oxford（牛津大学）；Microsoft（微软公司）；ETH Zurich（苏黎世联邦理工学院）；Princeton University（普林斯顿大学）；Harvard University（哈佛大学）
MIT Computer Science and Artificial Intelligence Laboratory（麻省理工学院计算机科学与人工智能实验室）
In partially observed environments, it can be useful for a human to provide the robot with declarative information that represents probabilistic relational constraints on properties of objects in the world, augmenting the robot's sensory observations. For instance, a robot tasked with a search-and-rescue mission may be informed by the human that two victims are probably in the same room. An important question arises: how should we represent the robot's internal knowledge so that this information is correctly processed and combined with raw sensory information? In this paper, we provide an effi…
This volume represents the proceedings of the 2nd Workshop on Robots in Clutter: Preparing robots for the real world, held June 27, 2013, at the Robotics: Science and Systems conference in Berlin, Germany.
We present a robotic setup for real-world testing and evaluation of human-robot and human-human collaborative learning. Leveraging the sample-efficiency of the Soft Actor-Critic algorithm, we have implemented a robotic platform able to learn a non-trivial collaborative task with a human partner, without pre-training in simulation, and using only 30 minutes of real-world interactions. This enables us to study Human-Robot and Human-Human collaborative learning through real-world interactions. We present preliminary results, showing that state-of-the-art deep learning methods can take human-robot…
University of Technology of Belfort-Montbéliard (UTBM)（贝尔福-蒙贝利亚尔技术大学（UTBM））；University of Sheffield（谢菲尔德大学）
This paper presents the research focus and ideas incorporated in the EPANer robotics team, entering the World Robot Challenge 2020 - Partner Robot Challenge (Real Space).
Robot world model representations are a vital part of robotic applications. However, there is no support for such representations in model-driven engineering tool chains. This work proposes a novel Domain Specific Language (DSL) for robotic world models that are based on the Robot Scene Graph (RSG) approach. The RSG-DSL can express (a) application specific scene configurations, (b) semantic scene structures and (c) inputs and outputs for the computational entities that are loaded into an instance of a world model.
This paper presents a practical approach towards implementing pathfinding algorithms on real-world and low-cost non- commercial hardware platforms. While using robotics simulation platforms as a test-bed for our algorithms we easily overlook real- world exogenous problems that are developed by external factors. Such problems involve robot wheel slips, asynchronous motors, abnormal sensory data or unstable power sources. The real-world dynamics tend to be very painful even for executing simple algorithms like a Wavefront planner or A-star search. This paper addresses designing techniques that t…
A Retrospective on the Robot Air Hockey Challenge: Benchmarking Robust, Reliable, and Safe Learning Techniques for Real-world Robotics
Intelligent Autonomous Systems, TU Darmstadt（达姆施塔特工业大学智能自主系统研究所）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Huawei German research center（华为德国研究中心）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；University College London（伦敦大学学院）；Idiap Research Institute（伊迪亚普研究所）；EPFL（洛桑联邦理工学院）；University of Luxembourg（卢森堡大学）；Karlsruher Institut für Technologie（卡尔斯鲁厄理工学院）；Universita di Bologna（博洛尼亚大学）；Politecnico di Milano（米兰理工大学）；Centre for Cognitive Science（认知科学中心）
The intuitive collaboration of humans and intelligent robots (embodied AI) in the real-world is an essential objective for many desirable applications of robotics. Whilst there is much research regarding explicit communication, we focus on how humans and robots interact implicitly, on motor adaptation level. We present a real-world setup of a human-robot collaborative maze game, designed to be non-trivial and only solvable through collaboration, by limiting the actions to rotations of two orthogonal axes, and assigning each axes to one player. This results in neither the human nor the agent be…
For robots to handle the numerous factors that can affect them in the real world, they must adapt to changes and unexpected events. Evolutionary robotics tries to solve some of these issues by automatically optimizing a robot for a specific environment. Most of the research in this field, however, uses simplified representations of the robotic system in software simulations. The large gap between performance in simulation and the real world makes it challenging to transfer the resulting robots to the real world. In this paper, we apply real world multi-objective evolutionary optimization to op…
The University of Tokyo（东京大学）；Matsuo Institute（松尾研究所）
Tidying up a household environment using a mobile manipulator poses various challenges in robotics, such as adaptation to large real-world environmental variations, and safe and robust deployment in the presence of humans.The Partner Robot Challenge in World Robot Challenge (WRC) 2020, a global competition held in September 2021, benchmarked tidying tasks in the real home environments, and importantly, tested for full system performances.For this challenge, we developed an entire household service robot system, which leverages a data-driven approach to adapt to numerous edge cases that occur d…
Reinforcement learning is a promising approach to developing hard-to-engineer adaptive solutions for complex and diverse robotic tasks. However, learning with real-world robots is often unreliable and difficult, which resulted in their low adoption in reinforcement learning research. This difficulty is worsened by the lack of guidelines for setting up learning tasks with robots. In this work, we develop a learning task with a UR5 robotic arm to bring to light some key elements of a task setup and study their contributions to the challenges with robots. We find that learning performance can be …
College of Engineering, Peking University（北京大学工学院）；Microsoft Research Asia（微软亚洲研究院）；National Engineering Research Center of Software Engineering, Peking University（北京大学国家软件工程研究中心）
In this paper, the circle formation control problem is addressed for a group of cooperative underactuated fish-like robots involving unknown nonlinear dynamics and disturbances. Based on the reinforcement learning and cognitive consistency theory, we propose a decentralized controller without the knowledge of the dynamics of the fish-like robots. The proposed controller can be transferred from simulation to reality. It is only trained in our established simulation environment, and the trained controller can be deployed to real robots without any manual tuning. Simulation results confirm that t…
Ritsumeikan University（立命馆大学）；Keio University（庆应义塾大学）；The University of Tokyo（东京大学）；Università Milano-Bicocca（米兰比可卡大学）；University of Essex（埃塞克斯大学）；Donders Institute for Brain, Cognition and Behaviour（唐德斯脑、认知与行为研究所）；Cajal International Neuroscience Center, Spanish National Research Council（西班牙国家研究委员会卡哈尔国际神经科学中心）；Bogazici University（博阿齐奇大学）；Queen Mary University of London（伦敦玛丽女王大学）；The University of Electro-Communications（电气通信大学）；National Autonomous University of Mexico（墨西哥国立自治大学）；Universidad Autónoma del Estado de Morelos（莫雷洛斯州自治大学）；Institute of Cognitive Sciences and Technologies, National Research Council of Italy（意大利国家研究委员会认知科学与技术研究所）
Creating autonomous robots that can actively explore the environment, acquire knowledge and learn skills continuously is the ultimate achievement envisioned in cognitive and developmental robotics. Their learning processes should be based on interactions with their physical and social world in the manner of human learning and cognitive development. Based on this context, in this paper, we focus on the two concepts of world models and predictive coding. Recently, world models have attracted renewed attention as a topic of considerable interest in artificial intelligence. Cognitive systems learn…
We present RCareWorld, a human-centric simulation world for physical and social robotic caregiving designed with inputs from stakeholders, including care recipients, caregivers, occupational therapists, and roboticists. RCareWorld has realistic human models of care recipients with mobility limitations and caregivers, home environments with multiple levels of accessibility and assistive devices, and robots commonly used for caregiving. It interfaces with various physics engines to model diverse material types necessary for simulating caregiving scenarios, and provides the capability to plan, co…
For a natural social human-robot interaction, it is essential for a robot to learn the human-like social skills. However, learning such skills is notoriously hard due to the limited availability of direct instructions from people to teach a robot. In this paper, we propose an intrinsically motivated reinforcement learning framework in which an agent gets the intrinsic motivation-based rewards through the action-conditional predictive model. By using the proposed method, the robot learned the social skills from the human-robot interaction experiences gathered in the real uncontrolled environmen…
Robotic agents that operate autonomously in the real world need to continuously explore their environment and learn from the data collected, with minimal human supervision. While it is possible to build agents that can learn in such a manner without supervision, current methods struggle to scale to the real world. Thus, we propose ALAN, an autonomously exploring robotic agent, that can perform tasks in the real world with little training and interaction time. This is enabled by measuring environment change, which reflects object movement and ignores changes in the robot position. We use this m…
