Deep reinforcement learning has proven to be a great success in allowing agents to learn complex tasks. However, its application to actual robots can be prohibitively expensive. Furthermore, the unpredictability of human behavior in human-robot interaction tasks can hinder convergence to a good policy. In this paper, we present an architecture that allows agents to learn models of stochastic environments and use them to accelerate learning. We descirbe how an environment model can be learned online and used to generate synthetic transitions, as well as how an agent can leverage these synthetic data to accelerate learning. We validate our approach using an experiment in which a robotic arm has to complete a task composed of a series of actions based on human gestures. Results show that our approach leads to significantly faster learning, requiring much less interaction with the environment. Furthermore, we demonstrate how learned models can be used by a robot to produce optimal plans in real world applications.
ImagineBench: Evaluating Reinforcement Learning with Large Language Model Rollouts
National Key Laboratory for Novel Software Technology Nanjing University China（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence Nanjing University China（南京大学人工智能学院）；The University of Hong Kong（香港大学）
The Hong Kong University of Science and Technology（香港科技大学）
Traditional model-based reinforcement learning (RL) methods generate forward rollout traces using the learnt dynamics model to reduce interactions with the real environment. The recent model-based RL method considers the way to learn a backward model that specifies the conditional probability of the previous state given the previous action and the current state to additionally generate backward rollout trajectories. However, in this type of model-based method, the samples derived from backward rollouts and those from forward rollouts are simply aggregated together to optimize the policy via th…
Learning from Hindsight for VLA Reinforcement Learning
Massachusetts Institute of Technology（麻省理工学院）；MIT-IBM Computing Research Lab（麻省理工学院-IBM计算研究实验室）；Stanford University（斯坦福大学）；University of California, San Diego（加利福尼亚大学圣地亚哥分校）
Learning robotic tasks in the real world is still highly challenging and effective practical solutions remain to be found. Traditional methods used in this area are imitation learning and reinforcement learning, but they both have limitations when applied to real robots. Combining reinforcement learning with pre-collected demonstrations is a promising approach that can help in learning control policies to solve robotic tasks. In this paper, we propose an algorithm that uses novel techniques to leverage offline expert data using offline and online training to obtain faster convergence and impro…
Yale University（耶鲁大学）；Technische Universität Darmstadt（达姆施塔特工业大学）
Substantial advancements to model-based reinforcement learning algorithms have been impeded by the model-bias induced by the collected data, which generally hurts performance. Meanwhile, their inherent sample efficiency warrants utility for most robot applications, limiting potential damage to the robot and its environment during training. Inspired by information theoretic model predictive control and advances in deep reinforcement learning, we introduce Model Predictive Actor-Critic (MoPAC), a hybrid model-based/model-free method that combines model predictive rollouts with policy optimizatio…
Bosch Center for Artificial Intelligence（博世人工智能中心）；University of Amsterdam（阿姆斯特丹大学）
Solving robotic navigation tasks via reinforcement learning (RL) is challenging due to their sparse reward and long decision horizon nature. However, in many navigation tasks, high-level (HL) task representations, like a rough floor plan, are available. Previous work has demonstrated efficient learning by hierarchal approaches consisting of path planning in the HL representation and using sub-goals derived from the plan to guide the RL policy in the source task. However, these approaches usually neglect the complex dynamics and sub-optimal sub-goal-reaching capabilities of the robot during pla…
Reinforcement learning (RL) has been shown to be effective at learning control from experience. However, RL typically requires a large amount of online interaction with the environment. This limits its applicability to real-world settings, such as in robotics, where such interaction is expensive. In this work we investigate ways to minimize online interactions in a target task, by reusing a suboptimal policy we might have access to, for example from training on related prior tasks, or in simulation. To this end, we develop two RL algorithms that can speed up training by using not only the acti…
A critical and re-configured HRI might look to the arts, where another history of robots has been unfolding since the Czech artist Karel Capek's critical robotic labor parable of 1921, in which the word robot was coined in its modern usage. This paper explores several vectors by which artist-created robots, both physical and imaginary, have offered pronounced contrasts to robots-as-usual, and offers directions as to how these more emancipated cousins might be useful to the field of HRI.
Prioritized Rollouts for Efficient World Model-based Vision-Language-Action Policy Optimization
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术全国重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；Cirquar Technologies；Mila – Quebec AI Institute（Mila – 魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）
VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents
National Key Laboratory for Novel Software Technology, Nanjing University, China \& School of Artificial Intelligence, Nanjing University, China；Computational Intelligence Center (CIC), School of Computer；Artificial Intelligence, Shandong Jianzhu University, Jinan, 250101, China
Task-Error Residual Learning for Real-Robot Five-Ball Juggling
Technical University of Darmstadt（达姆施塔特工业大学）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Hessian Center for Artificial Intelligence (hessian.AI)（黑森州人工智能中心）
GAMBIT: Learning to Plan Continuous Multi-Robot Trajectories
University of Cambridge（剑桥大学）；Technical University of Berlin（柏林工业大学）；National Institute of Advanced Industrial Science and Technology (AIST)（产业技术综合研究所）
Can Tabular Foundation Models Guide Exploration in Robot Policy Learning?
Department of Mechanical Engineering, Carnegie Mellon University（卡内基梅隆大学机械工程系）；Inria, Département d’informatique de l’ENS, CNRS, PSL Research University（法国国家科学研究中心（CNRS）、巴黎综合理工学院（ENS）和PSL研究大学的Inria）
University of Tennessee, Knoxville（田纳西大学诺克斯维尔分校）；University of Kentucky（肯塔基大学）；Ferdowsi University of Mashad（菲尔多西大学马什哈德分校）
Brain computer interface (BCI) provides promising applications in neuroprosthesis and neurorehabilitation by controlling computers and robotic devices based on the patient's intentions. Here, we have developed a novel BCI platform that controls a personalized social robot using noninvasively acquired brain signals. Scalp electroencephalogram (EEG) signals are collected from a user in real-time during tasks of imaginary movements. The imagined body kinematics are decoded using a regression model to calculate the user-intended velocity. Then, the decoded kinematic information is mapped to contro…
Ain Shams University（艾因沙姆斯大学）；Jacobs University（雅各布大学）；Stanford University（斯坦福大学）；KTH Royal Institute of Technology（KTH皇家理工学院）
Reinforcement learning (RL) is capable of sophisticated motion planning and control for robots in uncertain environments. However, state-of-the-art deep RL approaches typically lack safety guarantees, especially when the robot and environment models are unknown. To justify widespread deployment, robots must respect safety constraints without sacrificing performance. Thus, we propose a Black-box Reachability-based Safety Layer (BRSL) with three main components: (1) data-driven reachability analysis for a black-box robot model, (2) a trajectory rollout planner that predicts future actions and ob…
Georgia Institute of Technology（佐治亚理工学院）；Meta AI；Google Atlanta（谷歌亚特兰大）
We present Variable Experience Rollout (VER), a technique for efficiently scaling batched on-policy reinforcement learning in heterogenous environments (where different environments take vastly different times to generate rollouts) to many GPUs residing on, potentially, many machines. VER combines the strengths of and blurs the line between synchronous and asynchronous on-policy RL methods (SyncOnRL and AsyncOnRL, respectively). VER learns from on-policy experience (like SyncOnRL) and has no synchronization points (like AsyncOnRL). VER leads to significant and consistent speed-ups across a bro…
In the past few years, off-policy reinforcement learning methods have shown promising results in their application for robot control. Deep Q-learning, however, still suffers from poor data-efficiency and is susceptible to stochasticity in the environment or reward functions which is limiting with regard to real-world applications. We alleviate these problems by proposing two novel off-policy Temporal-Difference formulations: (1) Truncated Q-functions which represent the return for the first n steps of a target-policy rollout w.r.t. the full action-value and (2) Shifted Q-functions, acting as t…
Harvard University（哈佛大学）；Arizona State University（亚利桑那州立大学）
In this paper we consider infinite horizon discounted dynamic programming problems with finite state and control spaces, partial state observations, and a multiagent structure. We discuss and compare algorithms that simultaneously or sequentially optimize the agents' controls by using multistep lookahead, truncated rollout with a known base policy, and a terminal cost function approximation. Our methods specifically address the computational challenges of partially observable multiagent problems. In particular: 1) We consider rollout algorithms that dramatically reduce required computation whi…
Trust-Aware Sequential Decision Making and Rollout Planning for Resilient Multi-Robot Systems
School of Engineering and Applied Sciences, Harvard University（哈佛大学工程与应用科学学院）；Whiting School of Engineering at the Johns Hopkins University（约翰斯·霍普金斯大学惠廷工程学院）
SC3-Eval: Evaluating Robot Foundation Models via Self-Consistent Video Generation
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；NVIDIA（英伟达）；Physical Intelligence；Stanford University（斯坦福大学）；UC Berkeley（加州大学伯克利分校）；Allen Institute for AI（艾伦人工智能研究所）
University of New South Wales（新南威尔士大学）；Australian National University（澳大利亚国立大学）
The movement of cooperative robots in a densely cluttered environment may not be possible if the formation type is invariant. Hence, we investigate a new method for time-varying formation control for a group of heterogeneous autonomous vehicles, which may include Unmanned Ground Vehicles (UGV) and Unmanned Aerial Vehicles (UAV). We have extended a Negative-Imaginary (NI) consensus control approach to switch the formation shape of the robots whilst only using the relative distance between agents and between agents and obstacles. All agents can automatically create a new safe formation to overco…
University of New South Wales（新南威尔士大学）；Australian National University（澳大利亚国立大学）
This paper tackles the distributed leader-follower (L-F) control problem for heterogeneous mobile robots in unknown environments requiring obstacle avoidance, inter-robot collision avoidance, and reliable robot communications. To prevent an inter-robot collision, we employ a virtual propulsive force between robots. For obstacle avoidance, we present a novel distributed Negative-Imaginary (NI) variant formation tracking control approach and a dynamic network topology methodology which allows the formation to change its shape and the robot to switch their roles. In the case of communication or s…
WorldSample: Closed-loop Real-robot RL with World Modelling
PINE Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（南洋理工大学电气与电子工程学院PINE实验室）；Department of Electronic Engineering, Tsinghua University, Beijing, China（清华大学电子工程系）；School of Automation, Central South University, Changsha, China（中南大学自动化学院）；School of Automation, Beijing University of Posts and Telecommunications, Beijing, China（北京邮电大学自动化学院）
