This paper identifies and addresses the problems with naively combining (reinforcement) learning-based controllers and state estimators for robotic in-hand manipulation. Specifically, we tackle the challenging task of purely tactile, goal-conditioned, dextrous in-hand reorientation with the hand pointing downwards. Due to the limited sensing available, many control strategies that are feasible in simulation when having full knowledge of the object's state do not allow for accurate state estimation. Hence, separately training the controller and the estimator and combining the two at test time leads to poor performance. We solve this problem by coupling the control policy to the state estimator already during training in simulation. This approach leads to more robust state estimation and overall higher performance on the task while maintaining an interpretability advantage over end-to-end policy learning. With our GPU-accelerated implementation, learning from scratch takes a median training time of only 6.5 hours on a single, low-cost GPU. In simulation experiments with the DLR-Hand II and for four significantly different object shapes, we provide an in-depth analysis of the performance of our approach. We demonstrate the successful sim2real transfer by rotating the four objects to all 24 orientations in the $π/2$ discretization of SO(3), which has never been achieved for such a diverse set of shapes. Finally, our method can reorient a cube consecutively to nine goals (median), which was beyond the reach of previous methods in this challenging setting.
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Guangdong Institute of Intelligence Science and Technology(广东智能科学与技术研究院);Innovation and Research and Development Department, BoardWare Information System Company Ltd.(创新与研发部,BoardWare信息系统有限公司);School of Artificial Intelligence, Nanjing Agricultural University(人工智能学院,南京农业大学);School of Computing, National University of Singapore(计算机学院,新加坡国立大学);School of Electronic Engineering and Automation, Guilin University of Electronic Technology(电子工程与自动化学院,桂林电子科技大学)
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation
Hessian Centre for Artificial Intelligence(黑森人工智能中心);Technische Universität Dresden(德累斯顿工业大学);The Centre for Tactile Internet with Human-in-the-Loop (CeTI)(以人为中心的触觉互联网中心);Technical University of Darmstadt(达姆施塔特工业大学);German Research Center for AI (DFKI)(德国人工智能研究中心);Centre for Cognitive Science, TU Darmstadt(达姆施塔特工业大学认知科学中心)
Representation-Aligned Tactile Grounding for Contact-Rich Robotic Manipulation
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院智能信息处理上海市重点实验室);College of Intelligent Robotics and Advanced Manufacturing, Fudan University(复旦大学智能机器人与先进制造学院);Lenovo CTO Organization(联想首席技术官组织);Nanyang Technological University(南洋理工大学);TeleAl, China Telecom(中国电信天翼人工智能公司)
TacGNN:Learning Tactile-based In-hand Manipulation with a Blind Robot
Southern University of Science and Technology(南方科技大学);University of Hong Kong(香港大学);Tencent Robotics X(腾讯 Robotics X);University of Cambridge(剑桥大学);Imperial College London(帝国理工学院)
OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
TARS Robotics(TARS机器人实验室);National University of Singapore(新加坡国立大学);Fudan University(复旦大学);CASIA(中国科学院自动化研究所);Tsinghua University(清华大学);Zhongguancun Academy(中关村学院);Beihang University(北京航空航天大学)
Tactile-Conditioned Diffusion Policy for Force-Aware Robotic Manipulation
Department of Computer Science, Technical University of Darmstadt(德累斯顿技术大学计算机科学系);German Research Center for AI (DFKI)(德国人工智能研究中心);Centre for Cognitive Science, Technical University of Darmstadt(德累斯顿技术大学认知科学中心);Hessian Center for Artificial Intelligence (Hessian.AI), Darmstadt(黑森人工智能中心(Hessian.AI));Robotics Institute Germany (RIG)(德国机器人研究所)
TACTIC: Understanding Tactile Encoders and Conditioning for Contact-rich Robot Manipulation Policies
University of Technology Nuremberg(纽伦堡工业大学);Ludwig-Maximilians University Munich(慕尼黑路德维希-马克西米利安大学);Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Deggendorf Institute of Technology(德根多夫理工学院);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);University of Tromso(特罗姆瑟大学);DLR-German Aerospace Center(德国航空航天中心)
The Role of Touch: Towards Optimal Tactile Sensing Distribution in Anthropomorphic Hands for Dexterous In-Hand Manipulation
VisLab, Institute for Systems;Soft Robotics Lab National University of Singapore, Singapore;BRAIR Lab, BioRobotics Institute Scuola Superiore Sant'Anna, Pisa, Italia
FOTS: A Fast Optical Tactile Simulator for Sim2Real Learning of Tactile-motor Robot Manipulation Skills
School of Automation, Southeast University(东南大学自动化学院);Key Laboratory of Measurement and Control of CSE, Ministry of Education(东南大学复杂工程系统测量与控制教育部重点实验室);Department of Engineering, King’s College London(伦敦国王学院工程系)
TAMEn: Tactile-Aware Manipulation Engine for Closed-Loop Data Collection in Contact-Rich Tasks
Fudan University(复旦大学);Shanghai Innovation Institute(上海创新研究院);OpenDriveLab at The University of Hong Kong(香港大学OpenDriveLab);Shanghai Jiao Tong University(上海交通大学);East China University of Science and Technology(华东理工大学)
Unified Visual-Tactile-Action Modeling from Human Demonstrations for Dexterous Manipulation
Shanghai Jiao Tong University(上海交通大学);State Key Laboratory of General Artificial Intelligence, BIGAI(北京通用人工智能研究院 通用人工智能全国重点实验室);Sharpa Robotics;Beijing Institute of Technology(北京理工大学)
DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation
State Key Laboratory of Intelligent Manufacturing Equipment and Technology(智能制造装备与技术国家重点实验室);Huazhong University of Science and Technology(华中科技大学);Wuhan Huaweike Intelligent Technology Co., Ltd.(武汉华为凯科技有限公司)