In this paper, we explore generalizable, perception-to-action robotic manipulation for precise, contact-rich tasks. In particular, we contribute a framework for closed-loop robotic manipulation that automatically handles a category of objects, despite potentially unseen object instances and significant intra-category variations in shape, size and appearance. Previous approaches typically build a feedback loop on top of a real-time 6-DOF pose estimator. However, representing an object with a parameterized transformation from a fixed geometric template does not capture large intra-category shape variation. Hence we adopt the keypoint-based object representation proposed in kPAM for category-level pick-and-place, and extend it to closed-loop manipulation policies with contact-rich tasks. We first augment keypoints with local orientation information. Using the oriented keypoints, we propose a novel object-centric action representation in terms of regulating the linear/angular velocity or force/torque of these oriented keypoints. This formulation is surprisingly versatile -- we demonstrate that it can accomplish contact-rich manipulation tasks that require precision and dexterity for a category of objects with different shapes, sizes and appearances, such as peg-hole insertion for pegs and holes with significant shape variation and tight clearance. With the proposed object and action representation, our framework is also agnostic to the robot grasp pose and initial object configuration, making it flexible for integration and deployment.
MetaFold: Language-Guided Multi-Category Garment Folding Framework via Trajectory Generation and Foundation Model
School of Computing, National University of Singapore(新加坡国立大学计算机学院);NUS Guangzhou Research Translation and Innovation Institute(新加坡国立大学广州翻译与创新研究院);School of Computer Science, Nanjing University(南京大学计算机学院);School of Computer Science, Peking University(北京大学计算机学院);School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院)
GIFT: Geometry-Induced Functional Transfer for Category-level Object Manipulation
Extreme Robotics Laboratory, School of Metallurgy and Materials, University of Birmingham(伯明翰大学冶金与材料学院极端机器人实验室);Munich Institute of Robotics & Machine Intelligence, Technische Universität München (TUM)(慕尼黑工业大学机器人与人工智能研究所);School of Computer Science, University of Nottingham(诺丁汉大学计算机科学学院);Sorbonne Université, ISIR, Paris, France(巴黎法国索邦大学ISIR研究所)
PartDexTOG: Generating Dexterous Task-Oriented Grasping via Language-driven Part Analysis
College of Computer Science and Technology, National University of Defense Technology(计算机科学与技术学院,国防科技大学);College of Intelligence Science and Technology, National University of Defense Technology(智能科学与技术学院,国防科技大学)
ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));State Key Laboratory of Mobile Network and Mobile Multimedia Technology, Shenzhen(移动网络与移动多媒体技术国家重点实验室(深圳));ZTE Corporation(中兴通讯);Shanghai Jiao Tong University(上海交通大学);Central South University(中南大学)
Task-Aware Bimanual Affordance Prediction via VLM-Guided Semantic-Geometric Reasoning
Department of Computer Science, Technical University of Darmstadt(达姆施塔特工业大学计算机科学系);German Research Center for AI (DFKI)(德国人工智能研究中心);Centre for Cognitive Science, Technical University of Darmstadt(达姆施塔特工业大学认知科学中心);Hessian Center for Artificial Intelligence (Hessian.AI), Darmstadt(黑森州人工智能中心);Robotics Institute Germany (RIG)(德国机器人研究所)
FUNCanon: Learning Pose-Aware Action Primitives via Functional Object Canonicalization for Generalizable Robotic Manipulation
TAMS (Technical Aspects of Multimodal Systems), Department of Informatics, University of Hamburg(汉堡大学信息学院TAMS(多模态系统技术));Technical University of Munich(慕尼黑技术大学);Agile Robots SE(敏捷机器人有限公司)
Decoupled Object-Centric Video Understanding for Generating Robotic Manipulation Commands
School of Information Science, Japan Advanced Institute of Science and Technology(日本北陆先端科学技术大学院大学信息科学学院);University of Engineering and Technology, Vietnam National University(越南国立大学工程与技术大学);Department of Robotics, Hanyang University(汉阳大学机器人学系)
Towards Fine-Grained Object Manipulation: SAM3-Guided Visuomotor Policy with Persistent Memory Learning and Focused Visual Conditioning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);State Key Laboratory of Brain Cognition and Brain-Inspired Intelligence Technology(脑认知与类脑智能技术国家重点实验室);Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences(中国科学院脑科学与智能技术卓越创新中心)
Vi-TacMan: Articulated Object Manipulation via Vision and Touch
Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);School of Psychological and Cognitive Sciences, Peking University(北京大学心理与认知科学学院);State Key Lab of General AI, Peking University(北京大学通用人工智能国家重点实验室);Beijing Key Laboratory of Behavior and Mental Health, Peking University(北京大学行为与心理健康北京市重点实验室);LeapZenith AI Research(跃臻人工智能研究院);Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence(北京大学武汉人工智能研究院具身智能实验室)