Predictive models have been at the core of many robotic systems, from quadrotors to walking robots. However, it has been challenging to develop and apply such models to practical robotic manipulation due to high-dimensional sensory observations such as images. Previous approaches to learning models in the context of robotic manipulation have either learned whole image dynamics or used autoencoders to learn dynamics in a low-dimensional latent state. In this work, we introduce model-based prediction with self-supervised visual correspondence learning, and show that not only is this indeed possible, but demonstrate that these types of predictive models show compelling performance improvements over alternative methods for vision-based RL with autoencoder-type vision training. Through simulation experiments, we demonstrate that our models provide better generalization precision, particularly in 3D scenes, scenes involving occlusion, and in category-generalization. Additionally, we validate that our method effectively transfers to the real world through hardware experiments. Videos and supplementary materials available at https://sites.google.com/view/keypointsintothefuture
KGN-Pro: Keypoint-Based Grasp Prediction through Probabilistic 2D-3D Correspondence Learning
Zhejiang University（浙江大学）；Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；The State Key Laboratory of Industrial Control Technology（工业控制技术国家重点实验室）；China Research and Development Academy of Machinery Equipment（中国机械装备研究开发院）
End-to-end Reinforcement Learning of Robotic Manipulation with Robust Keypoints Representation
A*STAR Institute of High Performance Computing (IHPC)（新加坡科技研究局高性能计算研究所）；A*STAR Institute of Information Research (IR)（新加坡科技研究局信息研究所）；National University of Singapore (NUS)（新加坡国立大学）；Alibaba DAMO Academy（阿里巴巴达摩院）
Hunan University（湖南大学）；University of Alberta（阿尔伯塔大学）；Southern University of Science and Technology（南方科技大学）；Shaanxi Normal University（陕西师范大学）；The University of Western Australia（西澳大学）
AdaCropFollow: Self-Supervised Online Adaptation for Visual Under-Canopy Navigation
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Bonn（波恩大学）；University of Oxford（牛津大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）
