Humans have impressive generalization capabilities when it comes to manipulating objects and tools in completely novel environments. These capabilities are, at least partially, a result of humans having internal models of their bodies and any grasped object. How to learn such body schemas for robots remains an open problem. In this work, we develop an self-supervised approach that can extend a robot's kinematic model when grasping an object from visual latent representations. Our framework comprises two components: (1) we present a multi-modal keypoint detector: an autoencoder architecture trained by fusing proprioception and vision to predict visual key points on an object; (2) we show how we can use our learned keypoint detector to learn an extension of the kinematic chain by regressing virtual joints from the predicted visual keypoints. Our evaluation shows that our approach learns to consistently predict visual keypoints on objects in the manipulator's hand, and thus can easily facilitate learning an extended kinematic chain to include the object grasped in various configurations, from a few seconds of visual data. Finally we show that this extended kinematic chain lends itself for object manipulation tasks such as placing a grasped object and present experiments in simulation and on hardware.
Multi-Modal Manipulation via Multi-Modal Policy Consensus
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Columbia University（哥伦比亚大学）；Massachusetts Institute of Technology（麻省理工学院）；Harvard University（哈佛大学）
CLASP: General-Purpose Clothes Manipulation with Semantic Keypoints
School of Computing, Smart System Institute, National University of Singapore, Singapore（计算学院、智能系统研究所、新加坡国立大学，新加坡）；Department of Electronic and Electrical Engineering, Southern University of Science and Technology, China（电子与电气工程系、南方科技大学，中国）
End-to-end Reinforcement Learning of Robotic Manipulation with Robust Keypoints Representation
A*STAR Institute of High Performance Computing (IHPC)（新加坡科技研究局高性能计算研究所）；A*STAR Institute of Information Research (IR)（新加坡科技研究局信息研究所）；National University of Singapore (NUS)（新加坡国立大学）；Alibaba DAMO Academy（阿里巴巴达摩院）
3D Affordance Keypoint Detection for Robotic Manipulation
Advanced Robotics Centre, National University of Singapore（新加坡国立大学先进机器人中心）；Department of Mechanical Engineering, National University of Singapore（新加坡国立大学机械工程系）
SKT: Integrating State-Aware Keypoint Trajectories with Vision-Language Models for Robotic Garment Manipulation
Shanghai Jiao Tong University（上海交通大学）；The Hong Kong University of Science and Technology（香港科技大学）；National University of Singapore（新加坡国立大学）；Yuandao AI（元岛人工智能）；Shanghai AI Lab（上海人工智能实验室）
Synthetic Data Generation and Vision-based Wrinkle and Keypoint Detection for Bimanual Cloth Manipulation
Department of Engineering, University of Luxembourg（卢森堡大学工程系）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）；Université de Lorraine, Arts et Metiers Institute of Technology, LCFC（洛林大学，艺术与工艺技术学院，LCFC）
Robotic Manipulation Framework Based on Semantic Keypoints for Packing Shoes of Different Sizes, Shapes, and Softness
College of Mechanical and Electrical Engineering, Nanjing University of Aeronautics and Astronautics（机械与电子工程学院，南京航空航天大学）；State Key Laboratory of Internet of Things for Smart City（智能城市物联网国家重点实验室）；Department of Electromechanical Engineering, University of Macau（机电工程系，澳门大学）
AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoning
Wangxuan Institute of Computer Technology, Peking University（北京大学计算机技术研究院）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
On the Generalization Capabilities, Design Choices and Limitations of Keypoint Imitation Learning
AI and Robotics Lab, IDLAB-AIRO, Ghent University-imec（人工智能与机器人实验室，IDLAB-AIRO，根特大学-imec）；Robotics, Perception and Learning Lab, (RPL), EECS, KTH Royal Institute of Technology（机器人、感知与学习实验室，（RPL），EECS，皇家理工学院）；INCAR Robotics AB, Stockholm, Sweden（INCAR机器人公司，斯德哥尔摩，瑞典）
FUNCTO: Function-Centric One-Shot Imitation Learning for Tool Manipulation
Southern University of Science and Technology（南方科技大学）；National University of Singapore（新加坡国立大学）；Smart Systems Institute, National University of Singapore（新加坡国立大学智能系统研究院）
Multi-Keypoint Affordance Representation for Functional Dexterous Grasping
School of Artificial Intelligence and Robotics, Hunan University（人工智能与机器人学院，湖南大学）；College of Computer Science and Electronic Engineering, Hunan University（计算机科学与电子工程学院，湖南大学）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（机器人视觉感知与控制技术国家工程研究中心，湖南大学）
LEMMo-Plan: LLM-Enhanced Learning from Multi-Modal Demonstration for Planning Sequential Contact-Rich Manipulation Tasks
School of Computation, Information and Technology, Technical University of Munich（慕尼黑工业大学计算、信息与技术学院）；Aalto University（阿尔托大学）；Nanjing University (Suzhou Campus)（南京大学苏州校区）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
