We propose DINOBot, a novel imitation learning framework for robot manipulation, which leverages the image-level and pixel-level capabilities of features extracted from Vision Transformers trained with DINO. When interacting with a novel object, DINOBot first uses these features to retrieve the most visually similar object experienced during human demonstrations, and then uses this object to align its end-effector with the novel object to enable effective interaction. Through a series of real-world experiments on everyday tasks, we show that exploiting both the image-level and pixel-level properties of vision foundation models enables unprecedented learning efficiency and generalisation. Videos and code are available at https://www.robot-learning.uk/dinobot.
TypeTele: Releasing Dexterity in Teleoperation by Dexterous Manipulation Types
School of Computer Science and Engineering, Sun Yat-sen University, China(中山大学计算机科学与工程学院);Stanford University, USA(斯坦福大学);Imperial College London, UK(伦敦帝国理工学院)
VTouch++: A Multimodal Dataset with Vision-Based Tactile Enhancement for Bimanual Manipulation
Humanoid Robot (Shanghai) Co., Ltd.(人形机器人(上海)有限公司);Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University(上海智能自主系统研究院,同济大学);School of Astronautics, Harbin Institute of Technology(航天学院,哈尔滨工程大学)
Skill-Aware Diffusion for Generalizable Robotic Manipulation
School of Control Science and Engineering, Shandong University(控制科学与工程学院,山东大学);Department of Computer Science, The University of Manchester(计算机科学系,曼彻斯特大学)
Robust Bayesian Scene Reconstruction with Retrieval-Augmented Priors for Precise Grasping and Planning
University of Utah Robotics Center and Kahlert School of Computing, University of Utah(犹他大学机器人中心和Kahlert计算学院,犹他大学);College of Connected Computing, Vanderbilt(连接计算学院,范德比尔特大学);School of Computer Science, The University of Sydney(计算机科学学院,悉尼大学);Robotics Institute, Carnegie Mellon University(机器人研究所,卡内基梅隆大学);NVIDIA Corporation(NVIDIA公司)
Dynamic Resilient Spatio-Semantic Memory with Hybrid Localization for Mobile Manipulation
School of Mechanical Engineering and Automation, Beihang University(北京航空航天大学机械工程及自动化学院);Department of Systems Engineering, City University of Hong Kong(香港城市大学系统工程系);School of Computing, National University of Singapore(新加坡国立大学计算机学院)
RaycastGrasp: Eye-Gaze Interaction with Wearable Devices for Robotic Manipulation
College of Engineering Northeastern University Boston, USA;Khoury College of Computer Sciences Northeastern University Boston, USA;Department of Civil \& Environmental Engineering Northeastern University Boston, USA
Decompose and Recompose: Reasoning New Skills from Existing Abilities for Cross-Task Robotic Manipulation
School of Artificial Intelligence, College of Intelligence and Computing, Tianjin University, China(人工智能学院,智能与计算学院,天津大学,中国);School of Computer Science and Information Engineering, Hefei University of Technology, China(计算机科学与信息工程学院,合肥工业大学,中国)
本文提出Decompose and Recompose框架,通过分解现有任务演示为可解释的技能-动作对,实现跨任务机械操作的零样本泛化。
Language-Guided Grasping under Partial Observation for Mobile Manipulation in Field Inspection and Maintenance
Department of Mechanical Engineering, Federal University of Uberlândia(联邦大学伯南迪利亚机械工程系);Department of Mechanical Engineering, University of São Paulo(圣保罗大学机械工程系);Graduate School of Information Sciences, Tohoku University(东北大学信息科学研究生院);Faculdade Israelita de Ensino e Pesquisa Albert Einstein, Hospital Israelita Albert Einstein(艾伯特·爱因斯坦以色列教学与研究学院,艾伯特·爱因斯坦医院)
RAAP: Retrieval-Augmented Affordance Prediction with Cross-Image Action Alignment
School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University(东南大学计算机科学与工程学院、新一代人工智能技术与交叉应用重点实验室);Southeast University-Monash University Joint Graduate School, Southeast University(东南大学-蒙纳士大学联合研究生院);School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院)
Non-Markovian Long-Horizon Robot Manipulation via Keyframe Chaining
Tongji University(同济大学);University of Technology Sydney(技术悉尼大学);University of Electronic Science and Technology of China(电子科学与技术大学);Advanced Institute of Big Data(大数据先进研究院)
OmniVIC: A Self-Improving Variable Impedance Controller with Vision-Language In-Context Learning for Safe Robotic Manipulation
Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, Genoa, Italy(人机交互实验室,意大利理工学院,热那亚);Georgia Institute of Technology, Atlanta, USA(佐治亚理工学院,美国亚特兰大)
RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills
University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);Massachusetts Institute of Technology(麻省理工学院);National University of Singapore(新加坡国立大学);NVIDIA(英伟达);MIT-IBM Watson AI Lab(麻省理工-IBM沃森人工智能实验室)
RoboHarn-Evo: Evolving Hierarchical Physical Knowledge for Self-Improving Robotic Manipulation
Renmin University of China(中国人民大学);Tsinghua University(清华大学);Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China(数据工程与知识工程重点实验室);XYZ Embodied AI(XYZ具身智能公司);Engineering Research Center of Database and Business Intelligence, Beijing, China(数据库与商务智能工程研究中心);Beijing Academy of Artificial Intelligence (BAAI)(北京智源人工智能研究院)