We address the problem of (a) predicting the trajectory of an arm reaching motion, based on a few seconds of the motion's onset, and (b) leveraging this predictor to facilitate shared-control manipulation tasks, easing the cognitive load of the operator by assisting them in their anticipated direction of motion. Our novel intent estimator, dubbed the \emph{Robot Trajectron} (RT), produces a probabilistic representation of the robot's anticipated trajectory based on its recent position, velocity and acceleration history. Taking arm dynamics into account allows RT to capture the operator's intent better than other SOTA models that only use the arm's position, making it particularly well-suited to assist in tasks where the operator's intent is susceptible to change. We derive a novel shared-control solution that combines RT's predictive capacity to a representation of the locations of potential reaching targets. Our experiments demonstrate RT's effectiveness in both intent estimation and shared-control tasks. We will make the code and data supporting our experiments publicly available at https://github.com/mousecpn/Robot-Trajectron.git.
Honda Research Institute EU（本田欧洲研究院）；Honda R&D Co., Ltd. Innovative Research Excellence（本田研发株式会社创新卓越研究中心）
Shared control can help in teleoperated object manipulation by assisting with the execution of the user's intention. To this end, robust and prompt intention estimation is needed, which relies on behavioral observations. Here, an intention estimation framework is presented, which uses natural gaze and motion features to predict the current action and the target object. The system is trained and tested in a simulated environment with pick and place sequences produced in a relatively cluttered scene and with both hands, with possible hand-over to the other hand. Validation is conducted across di…
In this paper, we present the implementation details of a Virtual Reality (VR)-based teleoperation interface for moving a robotic manipulator. We propose an iterative human-in-the-loop design where the user sets the next task-space waypoint for the robot's end effector and executes the action on the physical robot before setting the next waypoints. Information from the robot's surroundings is provided to the user in two forms: as a point cloud in 3D space and a video stream projected on a virtual wall. The feasibility of the selected end effector pose is communicated to the user by the color o…
Istituto Italiano di Tecnologia（意大利技术研究院）；University of Naples Federico II（那不勒斯费德里科二世大学）
Manipulation in contrast to grasping is a trajectorial task that needs to use dexterous hands. Improving the dexterity of robot hands, increases the controller complexity and thus requires to use the concept of postural synergies. Inspired from postural synergies, this research proposes a new framework called kernelized synergies that focuses on the re-usability of the same subspace for precision grasping and dexterous manipulation. In this work, the computed subspace of postural synergies; parameterized by probabilistic movement primitives, is treated with kernel to preserve its grasping and …
BiRoAD: Learning Shared and Role-Adaptive Representations for Bimanual Manipulation
Peking University（北京大学）；PrimeBot Research Institute（PrimeBot研究院）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；UC Berkeley（加州大学伯克利分校）
Institut de Robòtica i Informàtica Industrial, CSIC-UPC（西班牙国家研究委员会-加泰罗尼亚理工大学工业机器人与信息学研究所）；Worcester Polytechnic Institute（伍斯特理工学院）；University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）
Benchmarking of robotic manipulations is one of the open issues in robotic research. An important factor that has enabled progress in this area in the last decade is the existence of common object sets that have been shared among different research groups. However, the existing object sets are very limited when it comes to cloth-like objects that have unique particularities and challenges. This paper is a first step towards the design of a cloth object set to be distributed among research groups from the robotics cloth manipulation community. We present a set of household cloth objects and rel…
PINT: A Unified Model with Shared Predictive Intent for Robot Manipulation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhongguancun Academy（中关村学院）；Fudan University（复旦大学）；Tsinghua University（清华大学）
LaST-HD: Learning Latent Physical Reasoning from Scalable Human Data for Robot Manipulation
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）；The Chinese University of Hong Kong（香港中文大学）；Simplexity Robotics；Aether Tech
Collective Learning for Unified Robotic Manipulation via Predictive Dynamics Adaptation
School of Computation, Information and Technology, Technical University of Munich（慕尼黑工业大学计算、信息与技术学院）；School of Science and Technology, Nanjing University (Suzhou Campus)（南京大学（苏州校区）科学技术学院）
Cornell University（康奈尔大学）；Stanford University（斯坦福大学）；Brain of Things, Inc.（Brain of Things公司）
There is a large variety of objects and appliances in human environments, such as stoves, coffee dispensers, juice extractors, and so on. It is challenging for a roboticist to program a robot for each of these object types and for each of their instantiations. In this work, we present a novel approach to manipulation planning based on the idea that many household objects share similarly-operated object parts. We formulate the manipulation planning as a structured prediction problem and learn to transfer manipulation strategy across different objects by embedding point-cloud, natural language, …
UniBiDex: A Unified Teleoperation Framework for Robotic Bimanual Dexterous Manipulation
Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）；Dongguan Key Laboratory of Intelligent Equipment and Smart Industry, School of Advanced Engineering, The Great Bay University（东莞智能装备与智能制造重点实验室，高级工程学院，大湾区大学）；Department of Mechanical Engineering, The Hong Kong Polytechnic University（香港理工大学机械工程系）；Guangdong Academy of Sciences（广东省科学院）
Cross-Embodiment Robot Manipulation via a Unified Hand Action Space
Intelligent Robotics and Vision Lab, University of Texas at Dallas（德克萨斯大学达拉斯分校智能机器人与视觉实验室）；Intelligent Robotics and Interactive Systems Lab, Arizona State University（亚利桑那州立大学智能机器人与交互系统实验室）
Robots deployed in many real-world settings need to be able to acquire new skills and solve new tasks over time. Prior works on planning with skills often make assumptions on the structure of skills and tasks, such as subgoal skills, shared skill implementations, or task-specific plan skeletons, which limit adaptation to new skills and tasks. By contrast, we propose doing task planning by jointly searching in the space of parameterized skills using high-level skill effect models learned in simulation. We use an iterative training procedure to efficiently generate relevant data to train such mo…
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）；Innovation and Research and Development Department, BoardWare Information System Company Ltd.（创新与研发部，BoardWare信息系统有限公司）；School of Artificial Intelligence, Nanjing Agricultural University（人工智能学院，南京农业大学）；School of Computing, National University of Singapore（计算机学院，新加坡国立大学）；School of Electronic Engineering and Automation, Guilin University of Electronic Technology（电子工程与自动化学院，桂林电子科技大学）
