In this work we investigate and demonstrate benefits of a Bayesian approach to imitation learning from multiple sensor inputs, as applied to the task of opening office doors with a mobile manipulator. Augmenting policies with additional sensor inputs, such as RGB + depth cameras, is a straightforward approach to improving robot perception capabilities, especially for tasks that may favor different sensors in different situations. As we scale multi-sensor robotic learning to unstructured real-world settings (e.g. offices, homes) and more complex robot behaviors, we also increase reliance on simulators for cost, efficiency, and safety. Consequently, the sim-to-real gap across multiple sensor modalities also increases, making simulated validation more difficult. We show that using the Variational Information Bottleneck (Alemi et al., 2016) to regularize convolutional neural networks improves generalization to held-out domains and reduces the sim-to-real gap in a sensor-agnostic manner. As a side effect, the learned embeddings also provide useful estimates of model uncertainty for each sensor. We demonstrate that our method is able to help close the sim-to-real gap and successfully fuse RGB and depth modalities based on understanding of the situational uncertainty of each sensor. In a real-world office environment, we achieve 96% task success, improving upon the baseline by +16%.
A Comparison of Imitation Learning Algorithms for Bimanual Manipulation
School of Computing and Augmented Intelligence at Arizona State University(亚利桑那州立大学计算与增强智能学院);TU Darmstadt(达姆施塔特工业大学);Robotics Institute at Carnegie Mellon University(卡内基梅隆大学机器人学院)
SPECI: Skill Prompts based Hierarchical Continual Imitation Learning for Robot Manipulation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学)
Dexterous Manipulation through Imitation Learning: A Survey
Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology and System(天津智能集群技术与系统重点实验室);School of Electrical and Information Engineering(电气与信息工程学院);Shandong University(山东大学);State Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);Division of Robotics, Perception and Learning(机器人、感知与学习 division);School of Control Science and Engineering(控制科学与工程学院);Massachusetts Institute of Technology(麻省理工学院);School of Computer Science(计算机科学学院);State Key Laboratory of Multimodal Artificial Intelligence Systems(多模态人工智能系统国家重点实验室);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence(人工智能学院)
Cloth Manipulation Using Random-Forest-Based Imitation Learning
University of Maryland at College Park(马里兰大学帕克分校);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);City University of Hong Kong(香港城市大学);the University of Hong Kong(香港大学)
Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challenges
Ecole Centrale de Lyon, CNRS, Universite Claude Bernard Lyon 1, INSA Lyon, Université Lumière Lyon 2, LIRIS, UMR5205(Lyon中央理工学院、CNRS、Claude Bernard Lyon 1大学、INSA Lyon、Lyon大学、LIRIS、UMR5205);Beihang University(北航大学);Dalian University of Technology(大连理工大学);Institut Universitaire de France (IUF)(法国大学研究院(IUF))
Imitation Learning for Adaptive Control of a Virtual Soft Exoglove
Department of Informatics, King’s College London(信息学院,伦敦国王学院);MyoLab;Harvard Medical School(哈佛医学院);Spaulding Rehabilitation Hospital(斯波林康复医院);Department of Bioengineering, Faculty of Engineering, Imperial College London(生物工程系,伦敦帝国理工学院工程学院);John A. Paulson School of Engineering and Applied Sciences, Harvard University(哈佛大学约翰·A·保罗森工程与应用科学学院)
RoboManipBaselines: A Unified Framework for Imitation Learning in Robotic Manipulation across Real and Simulation Environments
Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology (AIST)(日本国家先进工业科学技术研究院人工智能研究中心);CNRS-AIST JRL (Joint Robotics Laboratory), IRL(法国国家科学研究中心与日本AIST联合机器人实验室);Institute for AI and Robotics, Future Robotics Organization, Waseda University(早稻田大学未来机器人组织人工智能与机器人研究所);AI Robot Association (AIRoA)(人工智能机器人协会)
Gaze-Guided Task Decomposition for Imitation Learning in Robotic Manipulation
The University of Tokyo(东京大学);Graduate School of Information Science and Technology(信息科学与技术研究生院);Laboratory for Intelligent Systems and Informatics(智能系统与信息学实验室)
TACT: Humanoid Whole-body Contact Manipulation through Deep Imitation Learning with Tactile Modality
CNRS-AIST JRL (Joint Robotics Laboratory), IRL and National Institute of Advanced Industrial Science and Technology (AIST)(CNRS-AIST联合机器人实验室、IRL和国家先进工业科学与技术研究院);Tokyo University of Science(东京科学大学);University of Tsukuba(茨口大学)
Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulation
AI Thrust, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)AI Thrust);The Hong Kong University of Science and Technology(香港科技大学)