Learning to interact with the environment not only empowers the agent with manipulation capability but also generates information to facilitate building of action understanding and imitation capabilities. This seems to be a strategy adopted by biological systems, in particular primates, as evidenced by the existence of mirror neurons that seem to be involved in multi-modal action understanding. How to benefit from the interaction experience of the robots to enable understanding actions and goals of other agents is still a challenging question. In this study, we propose a novel method, deep modality blending networks (DMBN), that creates a common latent space from multi-modal experience of a robot by blending multi-modal signals with a stochastic weighting mechanism. We show for the first time that deep learning, when combined with a novel modality blending scheme, can facilitate action recognition and produce structures to sustain anatomical and effect-based imitation capabilities. Our proposed system, can be conditioned on any desired sensory/motor value at any time-step, and can generate a complete multi-modal trajectory consistent with the desired conditioning in parallel avoiding accumulation of prediction errors. We further showed that given desired images from different perspectives, i.e. images generated by the observation of other robots placed on different sides of the table, our system could generate image and joint angle sequences that correspond to either anatomical or effect based imitation behavior. Overall, the proposed DMBN architecture not only serves as a computational model for sustaining mirror neuron-like capabilities, but also stands as a powerful machine learning architecture for high-dimensional multi-modal temporal data with robust retrieval capabilities operating with partial information in one or multiple modalities.
TACT: Humanoid Whole-body Contact Manipulation through Deep Imitation Learning with Tactile Modality
CNRS-AIST JRL (Joint Robotics Laboratory), IRL and National Institute of Advanced Industrial Science and Technology (AIST)(CNRS-AIST联合机器人实验室、IRL和国家先进工业科学与技术研究院);Tokyo University of Science(东京科学大学);University of Tsukuba(茨口大学)
Prompting Multi-Modal Tokens to Enhance End-to-End Autonomous Driving Imitation Learning with LLMs
University of Technology Sydney(悉尼科技大学);Australia Artificial Intelligence Institute(澳大利亚人工智能研究院);School of Computer Science(计算机学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
CoLA-Flow Policy: Temporally Coherent Imitation Learning via Continuous Latent Action Flow Matching for Robotic Manipulation
State Key Laboratory of Robotics and System, Harbin Institute of Technology(机器人系统国家重点实验室,哈尔滨工业大学);The University of Sydney(悉尼大学);Honor Device Co., Ltd.(荣耀设备有限公司)
Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation
Interactive Robot Perception & Learning, TU Darmstadt(互动机器人感知与学习,图腾达姆施塔特大学);Intelligent Autonomous Systems, TU Darmstadt(智能自主系统,图腾达姆施塔特大学);Hessian AI(黑森人工智能);Robotics Institute Germany(德国机器人研究所)
PressMimic: Pressure-Guided Motion Capture and Control for Humanoid Robot Imitation
School of Electronic Science and Engineering, Nanjing University(南京大学电子科学与工程学院);Key Laboratory of Optoelectronic Devices and Systems with Extreme Performances of MOE, Nanjing University(南京大学极端性能光电技术与系统教育部重点实验室);BNRist, Tsinghua University(清华大学北京信息科学与技术国家研究中心)
Improving Robustness to Out-of-Distribution States in Imitation Learning via Deep Koopman-Boosted Diffusion Policy
Chair for Computer Aided Medical Procedures and Augmented Reality (CAMP), Technical University of Munich (TUM)(计算机辅助医疗程序与增强现实中心(CAMP),慕尼黑技术大学(TUM));Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Medical Intelligence and Robotic Cognition Lab (MIRoC), The University of Hong Kong (HKU)(医疗智能与机器人认知实验室(MIRoC),香港大学(HKU))
Evaluation Is All You Need for Multi-Modal Autonomous Driving
School of Vehicle and Mobility & College of AI, Tsinghua University(清华大学车辆与运载学院与人工智能学院);Dongfeng Motor Corporation Research & Development Institute(东风汽车集团有限公司研发总院)
MrGS: Multi-modal Radiance Fields with 3D Gaussian Splatting for RGB-Thermal Novel View Synthesis
Minnesota Robotics Institute (MnRI), University of Minnesota, Twin Cities(明尼苏达大学罗学院(MnRI)、明尼苏达大学双城分校);Department of Information Convergence Engineering (Artificial Intelligence Major), Pusan National University(信息融合工程系(人工智能专业),釜山国立大学);Department of Energy Engineering, Korea Institute of Energy Technology (KENTECH)(能源工程系,韩国能源技术研究所(KENTECH));School of Computer Science and Engineering, Pusan National University(计算机科学与工程学院,釜山国立大学)
Scalable Perception-Action-Communication Loops with Convolutional and Graph Neural Networks
Texas A&M University(德克萨斯农工大学);Rice University(莱斯大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Pennsylvania(宾夕法尼亚大学);US Army Research Laboratory(美国陆军研究实验室)