Best Paper Award of Cognitive Robotics in ICRA2021 IEEE Robotics and Automation Letters 2021, Proceedings of the 2021 International Conference on Robotics and Automation (ICRA 2021), 2021
Selection of appropriate tools and use of them when performing daily tasks is a critical function for introducing robots for domestic applications. In previous studies, however, adaptability to target objects was limited, making it difficult to accordingly change tools and adjust actions. To manipulate various objects with tools, robots must both understand tool functions and recognize object characteristics to discern a tool-object-action relation. We focus on active perception using multimodal sensorimotor data while a robot interacts with objects, and allow the robot to recognize their extrinsic and intrinsic characteristics. We construct a deep neural networks (DNN) model that learns to recognize object characteristics, acquires tool-object-action relations, and generates motions for tool selection and handling. As an example tool-use situation, the robot performs an ingredients transfer task, using a turner or ladle to transfer an ingredient from a pot to a bowl. The results confirm that the robot recognizes object characteristics and servings even when the target ingredients are unknown. We also examine the contributions of images, force, and tactile data and show that learning a variety of multimodal information results in rich perception for tool use.
Jacobs University Bremen（不来梅雅各布大学）；ShanghaiTech University（上海科技大学）
Underwater robot interventions require a high level of safety and reliability. A major challenge to address is a robust and accurate acquisition of localization estimates, as it is a prerequisite to enable more complex tasks, e.g. floating manipulation and mapping. State-of-the-art navigation in commercial operations, such as oil & gas production (OGP), rely on costly instrumentation. These can be partially replaced or assisted by visual navigation methods, especially in deep-sea scenarios where equipment deployment has high costs and risks. Our work presents a multimodal approach that adapts …
In this paper, we propose an active perception method for recognizing object categories based on the multimodal hierarchical Dirichlet process (MHDP). The MHDP enables a robot to form object categories using multimodal information, e.g., visual, auditory, and haptic information, which can be observed by performing actions on an object. However, performing many actions on a target object requires a long time. In a real-time scenario, i.e., when the time is limited, the robot has to determine the set of actions that is most effective for recognizing a target object. We propose an MHDP-based acti…
Humans usually perceive the world in a multimodal way that vision, touch, sound are utilised to understand surroundings from various dimensions. These senses are combined together to achieve a synergistic effect where the learning is more effectively than using each sense separately. For robotics, vision and touch are two key senses for the dexterous manipulation. Vision usually gives us apparent features like shape, color, and the touch provides local information such as friction, texture, etc. Due to the complementary properties between visual and tactile senses, it is desirable for us to co…
Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulation
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；School of Psychological and Cognitive Sciences, Peking University（北京大学心理与认知科学学院）；Beijing Key Lab of Behavior and Mental Health, Peking University（北京大学行为与心理健康北京市重点实验室）；Beijing Institute for General Artificial Intelligence（北京一般人工智能研究院）；State Key Lab for General Artificial Intelligence（一般人工智能国家重点实验室）；Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence（具身智能实验室，北京大学武汉人工智能研究院）；Department of Computer Science and Technology, University of Cambridge（剑桥大学计算机科学与技术系）
RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）；School of Engineering, Great Bay University（大湾区大学工程学院）；School of Computing and Information Technology, Great Bay University（大湾区大学计算与信息技术学院）
A General One-Shot Multimodal Active Perception Framework for Robotic Manipulation: Learning to Predict Optimal Viewpoint
Institute of Robotics and Automatic Information Systems, College of Artificial Intelligence, Nankai University（机器人与自动信息系统研究所，人工智能学院，南开大学）；Tianjin Key Laboratory of Intelligent Robotics, Nankai University（智能机器人重点实验室，南开大学）；College of Artificial Intelligence, Nankai University（人工智能学院，南开大学）；Department of Computing, The Hong Kong Polytechnic University（computing 部门，香港理工大学）
Colorado School of Mines（科罗拉多矿业学院）；Arizona State University（亚利桑那州立大学）
Apprenticeship learning has recently attracted a wide attention due to its capability of allowing robots to learn physical tasks directly from demonstrations provided by human experts. Most previous techniques assumed that the state space is known a priori or employed simple state representations that usually suffer from perceptual aliasing. Different from previous research, we propose a novel approach named Sequence-based Multimodal Apprenticeship Learning (SMAL), which is capable to simultaneously fusing temporal information and multimodal data, and to integrate robot perception with decisio…
YOLO with Kolmogorov-Arnold networks and vision-language foundation models for interpretable object detection with trustworthy multimodal AI in computer vision perception
University of Bath（巴斯大学）；Zhejiang University（浙江大学）
DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Models
MVSLab, Department of Mechanical and Mechatronics Engineering, University of Waterloo（滑铁卢大学机械与机电工程系MVSLab）；CompLING Lab, David R. Cheriton School of Computer Science, University of Waterloo（滑铁卢大学大卫·R·切里顿计算机科学学院CompLING Lab）；Department of Data and Systems Engineering, University of Hong Kong（香港大学数据与系统工程系）；Department of Mechanical Engineering, University of New Brunswick（新不伦瑞克大学机械工程系）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动性学院）
Real-time Vehicle-to-Vehicle Communication Based Network Cooperative Control System through Distributed Database and Multimodal Perception: Demonstrated in Crossroads
Zhejiang University（浙江大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Michigan（密歇根大学）
