Designing reinforcement learning (RL) problems that can produce delicate and precise manipulation policies requires careful choice of the reward function, state, and action spaces. Much prior work on applying RL to manipulation tasks has defined the action space in terms of direct joint torques or reference positions for a joint-space proportional derivative (PD) controller. In practice, it is often possible to add additional structure by taking advantage of model-based controllers that support both accurate positioning and control of the dynamic response of the manipulator. In this paper, we evaluate how the choice of action space for dynamic manipulation tasks affects the sample complexity as well as the final quality of learned policies. We compare learning performance across three tasks (peg insertion, hammering, and pushing), four action spaces (torque, joint PD, inverse dynamics, and impedance control), and using two modern reinforcement learning algorithms (Proximal Policy Optimization and Soft Actor-Critic). Our results lend support to the hypothesis that learning references for a task-space impedance controller significantly reduces the number of samples needed to achieve good performance across all tasks and algorithms.
Intelligent agents must be able to think fast and slow to perform elaborate manipulation tasks. Reinforcement Learning (RL) has led to many promising results on a range of challenging decision-making tasks. However, in real-world robotics, these methods still struggle, as they require large amounts of expensive interactions and have slow feedback loops. On the other hand, fast human-like adaptive control methods can optimize complex robotic interactions, yet fail to integrate multimodal feedback needed for unstructured tasks. In this work, we propose to factor the learning problem in a hierarc…
Volkswagen Group（大众集团）；Machine Learning Research Lab（机器学习研究实验室）
Multi-robot manipulation tasks involve various control entities that can be separated into dynamically independent parts. A typical example of such real-world tasks is dual-arm manipulation. Learning to naively solve such tasks with reinforcement learning is often unfeasible due to the sample complexity and exploration requirements growing with the dimensionality of the action and state spaces. Instead, we would like to handle such environments as multi-agent systems and have several agents control parts of the whole. However, decentralizing the generation of actions requires coordination acro…
Typical end-to-end formulations for learning robotic navigation involve predicting a small set of steering command actions (e.g., step forward, turn left, turn right, etc.) from images of the current state (e.g., a bird's-eye view of a SLAM reconstruction). Instead, we show that it can be advantageous to learn with dense action representations defined in the same domain as the state. In this work, we present "spatial action maps," in which the set of possible actions is represented by a pixel map (aligned with the input image of the current state), where each pixel represents a local navigatio…
Due to the broad range of applications of stochastic multi-armed bandit model, understanding the effects of adversarial attacks and designing bandit algorithms robust to attacks are essential for the safe applications of this model. In this paper, we introduce a new class of attack named action-manipulation attack. In this attack, an adversary can change the action signal selected by the user. We show that without knowledge of mean rewards of arms, our proposed attack can manipulate Upper Confidence Bound (UCB) algorithm, a widely used bandit algorithm, into pulling a target arm very frequentl…
KTH Royal Institute of Technology（瑞典皇家理工学院）；University of Salerno（萨莱诺大学）；University of Cassino and Southern Lazio（卡西诺大学和南拉齐奥大学）
We present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces such as manipulation of deformable objects. Planning is performed in a low-dimensional latent state space that embeds images. We define and implement a Latent Space Roadmap (LSR) which is a graph-based structure that globally captures the latent system dynamics. Our framework consists of two main components: a Visual Foresight Module (VFM) that generates a visual plan as a sequence of images, and an Action Proposal Network (APN) that predicts the actions between them. We show the …
KTH Royal Institute of Technology（KTH皇家理工学院）；Roma Tre University（罗马第三大学）；University of Cassino and Southern Lazio（卡西诺和南拉齐奥大学）
We present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces, focusing on manipulation of deformable objects. We propose a Latent Space Roadmap (LSR) for task planning which is a graph-based structure globally capturing the system dynamics in a low-dimensional latent space. Our framework consists of three parts: (1) a Mapping Module (MM) that maps observations given in the form of images into a structured latent space extracting the respective states as well as generates observations from the latent states, (2) the LSR which builds and conn…
Time-Unified Diffusion Policy with Action Discrimination for Robotic Manipulation
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家重点实验室）；National Engineering Research Center for Visual Information and Applications（视觉信息与应用国家工程研究中心）；Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University（西安交通大学人工智能与机器人研究院）；School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics（西南财经大学计算机与人工智能学院）
EDAR: Learning Environment-Dependent Action Representations for Robotic Manipulation
College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）；Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院智能信息处理上海市重点实验室）；TeleAI, China Telecom（中国电信天翼人工智能公司）
University of Maryland（马里兰大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
In this paper we present a formal computational framework for modeling manipulation actions. The introduced formalism leads to semantics of manipulation action and has applications to both observing and understanding human manipulation actions as well as executing them with a robotic mechanism (e.g. a humanoid robot). It is based on a Combinatory Categorial Grammar. The goal of the introduced framework is to: (1) represent manipulation actions with both syntax and semantic parts, where the semantic part employs $λ$-calculus; (2) enable a probabilistic semantic parsing schema to learn the $λ$-c…
DAM-VLA: A Dynamic Action Model-Based Vision-Language-Action Framework for Robot Manipulation
Advanced Research Lab, Samsung R&D Institute China-Beijing (SRCB)（三星研发中心中国北京先进研究实验室）；Samsung AI Center, DS Division（三星人工智能中心，DS部门）；Hanyang University ERICA（翰洋大学ERICA）
Rochester Institute of Technology（罗切斯特理工学院）；OSARO Inc（OSARO公司）；Siemens Corporation, Corporate Technology（西门子公司企业技术部）
In this work, we focus on multi-step manipulation tasks that involve long-horizon planning and considers progress reversal. Such tasks interlace high-level reasoning that consists of the expected states that can be attained to achieve an overall task and low-level reasoning that decides what actions will yield these states. We propose a sample efficient Previous Action Conditioned Robotic Manipulation Network (PAC-RoManNet) to learn the action-value functions and predict manipulation action candidates from visual observation of the scene and action-value predictions of the previous action. We …
In this paper, we propose a new action planning approach to automatically pack long linear elastic objects into common-size boxes with a bimanual robotic system. For that, we developed a hybrid geometric model to handle large-scale occlusions combining an online vision-based method and an offline reference template. Then, a reference point generator is introduced to automatically plan the reference poses for the predesigned action primitives. Finally, an action planner integrates these components enabling the execution of high-level behaviors and the accomplishment of packing manipulation task…
Object manipulation actions represent an important share of the Activities of Daily Living (ADLs). In this work, we study how to enable service robots to use human multi-modal data to understand object manipulation actions, and how they can recognize such actions when humans perform them during human-robot collaboration tasks. The multi-modal data in this study consists of videos, hand motion data, applied forces as represented by the pressure patterns on the hand, and measurements of the bending of the fingers, collected as human subjects performed manipulation actions. We investigate two dif…
Learning from demonstrations is a promising paradigm for transferring knowledge to robots. However, learning mobile manipulation tasks directly from a human teacher is a complex problem as it requires learning models of both the overall task goal and of the underlying actions. Additionally, learning from a small number of demonstrations often introduces ambiguity with respect to the intention of the teacher, making it challenging to commit to one model for generalizing the task to new settings. In this paper, we present an approach to learning flexible mobile manipulation action models and tas…
The object manipulation is a crucial ability for a service robot, but it is hard to solve with reinforcement learning due to some reasons such as sample efficiency. In this paper, to tackle this object manipulation, we propose a novel framework, AP-NPQL (Non-Parametric Q Learning with Action Primitives), that can efficiently solve the object manipulation with visual input and sparse reward, by utilizing a non-parametric policy for reinforcement learning and appropriate behavior prior for the object manipulation. We evaluate the efficiency and the performance of the proposed AP-NPQL for four ob…
Cross-Embodiment Robot Manipulation via a Unified Hand Action Space
Intelligent Robotics and Vision Lab, University of Texas at Dallas（德克萨斯大学达拉斯分校智能机器人与视觉实验室）；Intelligent Robotics and Interactive Systems Lab, Arizona State University（亚利桑那州立大学智能机器人与交互系统实验室）
Institute of Industrial and Control Engineering (IOC)（工业与控制工程研究所）；Universitat Politcnica de Catalunya (UPC) – Barcelona Tech（加泰罗尼亚理工大学 – 巴塞罗那科技大学）
Robotic manipulation involves actions where contacts occur between the robot and the objects. In this scope, the availability of physics-based engines allows motion planners to comprise dynamics between rigid bodies, which is necessary for planning this type of actions. However, physics-based motion planning is computationally intensive due to the high dimensionality of the state space and the need to work with a low integration step to find accurate solutions. On the other hand, manipulation actions change the environment and conditions further actions and motions. To cope with this issue, th…
UniWAM Technical Report: Unified Mobile Manipulation via Mixed-Stream World-Action Modeling and Manipulation Anchor Pose Supervision
Physical Superintelligence Lab, Fysics AI（Fysics AI 物理超级智能实验室）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）
OSARO Inc（OSARO公司）；Rochester Institute of Technology（罗切斯特理工学院）；Siemens Corporation（西门子公司）
Multi-step manipulation tasks in unstructured environments are extremely challenging for a robot to learn. Such tasks interlace high-level reasoning that consists of the expected states that can be attained to achieve an overall task and low-level reasoning that decides what actions will yield these states. We propose a model-free deep reinforcement learning method to learn multi-step manipulation tasks. We introduce a Robotic Manipulation Network (RoManNet), which is a vision-based model architecture, to learn the action-value functions and predict manipulation action candidates. We define a …
STARRY: Spatial-Temporal Action-Centric World Modeling for Robotic Manipulation
Beijing Institute of Technology（北京理工大学）；Zhongguancun Academy（中关村学院）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；University of Science and Technology of China（中国科学技术大学）；Harbin Institute of Technology（哈尔滨工业大学）；East China Normal University（华东师范大学）；DeepCybo
