Adapt2Reward: Adapting Video-Language Models to Generalizable Robotic Rewards via Failure Prompts
School of Software Technology, Zhejiang University（浙江大学软件学院）；School of Computer Sciene and Technology, Hangzhou Dianzi University（杭州电子科技大学计算机学院）；China Mobile (Zhejiang) Research & Innovation Institute（中国移动（浙江）研究院）；State Key Lab of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）；The Hong Kong Polytechnic University（香港理工大学）；Zhiyuan Research Institute（智源研究院）；School of Computer Sciene and Technology, Xidian University（西安电子科技大学计算机学院）
For a general-purpose robot to operate in reality, executing a broad range of instructions across various environments is imperative. Central to the reinforcement learning and planning for such robotic agents is a generalizable reward function. Recent advances in vision-language models, such as CLIP, have shown remarkable performance in the domain of deep learning, paving the way for open-domain visual recognition. However, collecting data on robots executing various language instructions across multiple environments remains a challenge. This paper aims to transfer video-language models with robust generalization into a generalizable language-conditioned reward function, only utilizing robot video data from a minimal amount of tasks in a singular environment. Unlike common robotic datasets used for training reward functions, human video-language datasets rarely contain trivial failure videos. To enhance the model's ability to distinguish between successful and failed robot executions, we cluster failure video features to enable the model to identify patterns within. For each cluster, we integrate a newly trained failure prompt into the text encoder to represent the corresponding failure mode. Our language-conditioned reward function shows outstanding generalization to new environments and new instructions for robot planning and reinforcement learning.
FUNCanon: Learning Pose-Aware Action Primitives via Functional Object Canonicalization for Generalizable Robotic Manipulation
TAMS (Technical Aspects of Multimodal Systems), Department of Informatics, University of Hamburg（汉堡大学信息学院TAMS（多模态系统技术））；Technical University of Munich（慕尼黑技术大学）；Agile Robots SE（敏捷机器人有限公司）
Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations
Carnegie Mellon University（卡内基梅隆大学）；University of Southern California（南加州大学）；University of California, Berkeley（加州大学伯克利分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Arizona State University（亚利桑那州立大学）；Adobe Systems, Inc.（Adobe系统公司）
We study the problem of learning a generalizable action policy for an intelligent agent to actively approach an object of interest in an indoor environment solely from its visual inputs. While scene-driven or recognition-driven visual navigation has been widely studied, prior efforts suffer severely from the limited generalization capability. In this paper, we first argue the object searching task is environment dependent while the approaching ability is general. To learn a generalizable approaching policy, we present a novel solution dubbed as GAPLE which adopts two channels of visual feature…
University of Oxford（牛津大学）；Hong Kong Polytechnic University（香港理工大学）
Recent works in robotic manipulation through reinforcement learning (RL) or imitation learning (IL) have shown potential for tackling a range of tasks e.g., opening a drawer or a cupboard. However, these techniques generalize poorly to unseen objects. We conjecture that this is due to the high-dimensional action space for joint control. In this paper, we take an alternative approach and separate the task of learning 'what to do' from 'how to do it' i.e., whole-body control. We pose the RL problem as one of determining the skill dynamics for a disembodied virtual manipulator interacting with ar…
SAMamba3D: adapting Segment Anything for generalizable 3D segmentation of multiphase pore-scale images
Department of Earth Science and Engineering, Imperial College London（帝国理工学院伦敦地球科学与工程系）；College of Artificial Intelligence, China University of Petroleum (Beijing)（中国石油大学（北京）人工智能学院）；College of Petroleum Engineering, China University of Petroleum (Beijing)（中国石油大学（北京）石油工程学院）
KTH Royal Institute of Technology（瑞典皇家理工学院）；Aalto University（阿尔托大学）
Deep reinforcement learning (RL) has enabled training action-selection policies, end-to-end, by learning a function which maps image pixels to action outputs. However, it's application to visuomotor robotic policy training has been limited because of the challenge of large-scale data collection when working with physical hardware. A suitable visuomotor policy should perform well not just for the task-setup it has been trained for, but also for all varieties of the task, including novel objects at different viewpoints surrounded by task-irrelevant objects. However, it is impractical for a robot…
Beijing Institute of General Artificial Intelligence (BIGAI)（北京通用人工智能研究院）；Tsinghua University（清华大学）；Peking University（北京大学）
Generating dexterous grasping has been a long-standing and challenging robotic task. Despite recent progress, existing methods primarily suffer from two issues. First, most prior arts focus on a specific type of robot hand, lacking the generalizable capability of handling unseen ones. Second, prior arts oftentimes fail to rapidly generate diverse grasps with a high success rate. To jointly tackle these challenges with a unified solution, we propose GenDexGrasp, a novel hand-agnostic grasping algorithm for generalizable grasping. GenDexGrasp is trained on our proposed large-scale multi-hand gra…
Learning Generalizable 3D Manipulation With 10 Demonstrations
State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所 机器人学国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；College of Automation Science and Engineering, South China University of Technology（华南理工大学自动化科学与工程学院）
Budapest University of Technology and Economics（布达佩斯技术与经济大学）
Imitation Learning uses the demonstrations of an expert to uncover the optimal policy and it is suitable for real-world robotics tasks as well. In this case, however, the training of the agent is carried out in a simulation environment due to safety, economic and time constraints. Later, the agent is applied in the real-life domain using sim-to-real methods. In this paper, we apply Imitation Learning methods that solve a robotics task in a simulated environment and use transfer learning to apply these solutions in the real-world environment. Our task is set in the Duckietown environment, where…
We are motivated by the goal of generalist robots that can complete a wide range of tasks across many environments. Critical to this is the robot's ability to acquire some metric of task success or reward, which is necessary for reinforcement learning, planning, or knowing when to ask for help. For a general-purpose robot operating in the real world, this reward function must also be able to generalize broadly across environments, tasks, and objects, while depending only on on-board sensor observations (e.g. RGB images). While deep learning on large and diverse datasets has shown promise as a …
Generalizable Long-Horizon Manipulations with Large Language Models
National University of Singapore（新加坡国立大学）；UC Berkeley（加州大学伯克利分校）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Towards Learning a Generalizable 3D Scene Representation from 2D Observations
University of Hamburg - Department of Informatics（汉堡大学信息学院）；ZAL Center of Applied Aeronautical Research（应用航空研究中心）；Hamburger Informatik Technologie-Center e.V. (HITeC)（汉堡信息科技中心（HITeC））
Peking University（北京大学）；Stanford University（斯坦福大学）；Tencent AI Lab（腾讯人工智能实验室）
Part assembly is a typical but challenging task in robotics, where robots assemble a set of individual parts into a complete shape. In this paper, we develop a robotic assembly simulation environment for furniture assembly. We formulate the part assembly task as a concrete reinforcement learning problem and propose a pipeline for robots to learn to assemble a diverse set of chairs. Experiments show that when testing with unseen chairs, our approach achieves a success rate of 74.5% under the object-centric setting and 50.0% under the full setting. We adopt an RRT-Connect algorithm as the baseli…
Robotic manipulation in complex open-world scenarios requires both reliable physical manipulation skills and effective and generalizable perception. In this paper, we propose a method where general purpose pretrained visual models serve as an object-centric prior for the perception system of a learned policy. We devise an object-level attentional mechanism that can be used to determine relevant objects from a few trajectories or demonstrations, and then immediately incorporate those objects into a learned policy. A task-independent meta-attention locates possible objects in the scene, and a ta…
Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis
School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；KEENON Robotics Co., Ltd.（科语机器人有限公司）；Suzhou Silicon Era Intelligent Technology Co., Ltd.（苏州硅时代智能科技有限公司）；College of Materials, Xiamen University（厦门大学材料学院）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；ByteDance（字节跳动）；State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院通用人工智能国家重点实验室）；Hubei Automation Institute（湖北省自动化研究所）
UC Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）；University of Pennsylvania（宾夕法尼亚大学）；CMU（卡内基梅隆大学）
Robot learning has emerged as a promising tool for taming the complexity and diversity of the real world. Methods based on high-capacity models, such as deep networks, hold the promise of providing effective generalization to a wide range of open-world environments. However, these same methods typically require large amounts of diverse training data to generalize effectively. In contrast, most robotic learning experiments are small-scale, single-domain, and single-robot. This leads to a frequent tension in robotic learning: how can we learn generalizable robotic controllers without having to c…
University of California, Berkeley（加州大学伯克利分校）；Boston University（波士顿大学）
Real-world robotics problems often occur in domains that differ significantly from the robot's prior training environment. For many robotic control tasks, real world experience is expensive to obtain, but data is easy to collect in either an instrumented environment or in simulation. We propose a novel domain adaptation approach for robot perception that adapts visual representations learned on a large easy-to-obtain source dataset (e.g. synthetic images) to a target real-world domain, without requiring expensive manual data annotation of real world data before policy search. Supervised domain…
A Generalizable Feature Extractor for Alzheimer's-Related Brain MRI Tasks
McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University（麦吉尔大学蒙特利尔神经学研究所麦康奈尔脑成像中心）；Department of Biomedical Engineering, McGill University（麦吉尔大学生物医学工程系）
G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulation
The University of Hong Kong（香港大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（人工智能研究所（TeleAI），中国电信）；Shenzhen University（深圳大学）；AgileX Robotics（AgileX机器人）；GDIIST；HKU Shanghai Intelligent Computing Research Center（香港大学上海智能计算研究中心）
The University of Tokyo（东京大学）；Google Research（谷歌研究院）
Domain generalization (DG) is a difficult transfer learning problem aiming to learn a generalizable model for unseen domains. Recent foundation models (FMs) are robust to many distribution shifts and, therefore, should substantially improve the performance of DG. In this work, we study generic ways to adopt CLIP, a Visual-Language Foundation Model, for DG problems in image classification. While ERM greatly improves the accuracy with bigger backbones and training datasets using standard DG benchmarks, fine-tuning FMs is not practical in many real-world situations. We propose Domain Prompt Learn…
Zero-shot Generalizable Incremental Learning for Vision-Language Object Detection
Institute of Automation, Chinese Academy of Sciences (CAS)（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Science, UCAS（中国科学院大学人工智能学院）；Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属第六人民医院）
Google Research, Robotics at Google Team（谷歌研究院，谷歌机器人团队）；University of Southern California（南加州大学）；University of California, Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）
One of the great promises of robot learning systems is that they will be able to learn from their mistakes and continuously adapt to ever-changing environments. Despite this potential, most of the robot learning systems today are deployed as a fixed policy and they are not being adapted after their deployment. Can we efficiently adapt previously learned behaviors to new environments, objects and percepts in the real world? In this paper, we present a method and empirical evidence towards a robot learning framework that facilitates continuous adaption. In particular, we demonstrate how to adapt…
