In this work, we present a multimodal system for active robot-object interaction using laser-based SLAM, RGBD images, and contact sensors. In the object manipulation task, the robot adjusts its initial pose with respect to obstacles and target objects through RGBD data so it can perform object grasping in different configuration spaces while avoiding collisions, and updates the information related to the last steps of the manipulation process using the contact sensors in its hand. We perform a series of experiment to evaluate the performance of the proposed system following the the RoboCup2018 international competition regulations. We compare our approach with a number of baselines, namely a no-feedback method and visual-only and tactile-only feedback methods, where our proposed visual-and-tactile feedback method performs best.
Language and Sketching: An LLM-driven Interactive Multimodal Multitask Robot Navigation Framework
ShanghaiTech University（上海科技大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Shanghai for Science and Technology（上海理工大学）；The University of Manchester（曼彻斯特大学）；University College London（伦敦大学学院）
We present a new interface for controlling a navigation robot in novel environments using coordinated gesture and language. We use a TurtleBot3 robot with a LIDAR and a camera, an embodied simulation of what the robot has encountered while exploring, and a cross-platform bridge facilitating generic communication. A human partner can deliver instructions to the robot using spoken English and gestures relative to the simulated environment, to guide the robot through navigation tasks.
Our overall program objective is to provide more natural ways for soldiers to interact and communicate with robots, much like how soldiers communicate with other soldiers today. We describe how the Wizard-of-Oz (WOz) method can be applied to multimodal human-robot dialogue in a collaborative exploration task. While the WOz method can help design robot behaviors, traditional approaches place the burden of decisions on a single wizard. In this work, we consider two wizards to stand in for robot navigation and dialogue management software components. The scenario used to elicit data is one in whi…
The CREATE database is composed of 14 hours of multimodal recordings from a mobile robotic platform based on the iRobot Create. The various sensors cover vision, audition, motors and proprioception. The dataset has been designed in the context of a mobile robot that can learn multimodal representations of its environment, thanks to its ability to navigate the environment. This ability can also be used to learn the dependencies and relationships between the different modalities of the robot (e.g. vision, audition), as they reflect both the external environment and the internal state of the robo…
University of Illinois Chicago（伊利诺伊大学芝加哥分校）；California State University Channel Islands（加州州立大学海峡群岛分校）
We outline our work on evaluating robots that assist older adults by engaging with them through multiple modalities that include physical interaction. Our thesis is that to increase the effectiveness of assistive robots: 1) robots need to understand and effect multimodal actions, 2) robots should not only react to the human, they need to take the initiative and lead the task when it is necessary. We start by briefly introducing our proposed framework for multimodal interaction and then describe two different experiments with the actual robots. In the first experiment, a Baxter robot helps a hu…
Colorado School of Mines（科罗拉多矿业学院）；U.S. Army Research Laboratory（美国陆军研究实验室）
Multi-robot systems of increasing size and complexity are used to solve large-scale problems, such as area exploration and search and rescue. A key decision in human-robot teaming is dividing a multi-robot system into teams to address separate issues or to accomplish a task over a large area. In order to address the problem of selecting teams in a multi-robot system, we propose a new multimodal graph embedding method to construct a unified representation that fuses multiple information modalities to describe and divide a multi-robot system. The relationship modalities are encoded as directed g…
ERR@HRI 2024 Challenge: Multimodal Detection of Errors and Failures in Human-Robot Interactions
Politecnico di Milano（米兰理工大学）；University of Cambridge（剑桥大学）；Cornell University（康奈尔大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Cankaya University（恰卡亚大学）；Cornell Tech（康奈尔科技学院）
State University of New York at Binghamton（纽约州立大学宾汉姆顿分校）；Tufts University（塔夫茨大学）
Robots frequently need to perceive object attributes, such as "red," "heavy," and "empty," using multimodal exploratory actions, such as "look," "lift," and "shake." Robot attribute learning algorithms aim to learn an observation model for each perceivable attribute given an exploratory action. Once the attribute models are learned, they can be used to identify attributes of new objects, answering questions, such as "Is this object red and empty?" Attribute learning and identification are being treated as two separate problems in the literature. In this paper, we first define a new problem cal…
Towards Language-Based Modulation of Assistive Robots through Multimodal Models
Center for Information and Language Processing (CIS)（信息与语言处理中心（CIS））；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））；Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与机器智能研究所（MIRMI））；Technical University Munich (TUM)（慕尼黑工业大学（TUM））
Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）
Gaze stabilization is fundamental for humanoid robots. By stabilizing vision, it enhances perception of the environment and keeps points of interest in the field of view. In this contribution, a multimodal gaze stabilization combining classic inverse kinematic control with vestibulo-ocular and optokinetic reflexes is introduced. Inspired by neuroscience, it implements a forward model that can modulate the reflexes based on the reafference principle. This principle filters self-generated movements out of the reflexive feedback loop. The versatility and effectiveness of this method are experimen…
Feedback design is an important aspect of person-following robots for older adults. This paper presents a user-centred design approach to ensure the design is focused on the needs and preferences of the users. A sequence of user studies with a total of 35 older adults (aged 62 years and older) was conducted to explore their preferences regarding feedback parameters for a socially assistive person-following robot. The preferred level of robot transparency and the desired content for the feedback was first explored. This was followed by an assessment of the preferred mode and timing of feedback.…
Feasibility-aware Imitation Learning from Observation with Multimodal Feedback
Division of Information Science, Graduate School of Science and Technology, Nara Institute of Science and Technology (NAIST)（信息科学系，科学技术研究生院，科学与技术国立研究所（NAIST））
Advantages of Multimodal versus Verbal-Only Robot-to-Human Communication with an Anthropomorphic Robotic Mock Driver
Centre for Applied Autonomous Sensor Systems (AASS)（应用自主传感器系统中心（AASS））；Örebro University（厄勒布鲁大学）；Robert Bosch GmbH（罗伯特·博世有限公司）；University of Skövde（舍夫德大学）；TU Munich（慕尼黑工业大学）
Wearable robotic hand rehabilitation devices can allow greater freedom and flexibility than their workstation-like counterparts. However, the field is generally lacking effective methods by which the user can operate the device: such controls must be effective, intuitive, and robust to the wide range of possible impairment patterns. Even when focusing on a specific condition, such as stroke, the variety of encountered upper limb impairment patterns means that a single sensing modality, such as electromyography (EMG), might not be sufficient to enable controls for a broad range of users. To add…
BronchoCopilot: Towards Autonomous Robotic Bronchoscopy via Multimodal Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
The interaction of Older adults with robots requires effective feedback to keep them aware of the state of the interaction for optimum interaction quality. This study examines the effect of different feedback modalities in a table setting robot assistant for elder care. Two different feedback modalities (visual and auditory) and their combination were evaluated for three complexity levels. The visual feedback included the use of LEDs and a GUI screen. The auditory feedback included alerts (beeps) and verbal commands. The results revealed that the quality of interaction was influenced mainly by…
Oregon State University（俄勒冈州立大学）；Rhodes College（罗德学院）
In this paper we investigate the influence interfaces and feedback have on human-robot trust levels when operating in a shared physical space. The task we use is specifying a "no-go" region for a robot in an indoor environment. We evaluate three styles of interface (physical, AR, and map-based) and four feedback mechanisms (no feedback, robot drives around the space, an AR "fence", and the region marked on the map). Our evaluation looks at both usability and trust. Specifically, if the participant trusts that the robot "knows" where the no-go region is and their confidence in the robot's abili…
Human emotions are expressed through multiple modalities, including verbal and non-verbal information. Moreover, the affective states of human users can be the indicator for the level of engagement and successful interaction, suitable for the robot to use as a rewarding factor to optimize robotic behaviors through interaction. This study demonstrates a multimodal human-robot interaction (HRI) framework with reinforcement learning to enhance the robotic interaction policy and personalize emotional interaction for a human user. The goal is to apply this framework in social scenarios that can let…
The problem of distributed formation control of nonholonomic mobile robots is addressed in this paper, in which the robots are designed to track a formation. Collision avoidance among agents is guaranteed using a control law based on a repulsive force. In an uncertain environment where obstacles exist, the construction of repulsive force and rotational direction enables agents to avoid and pass the obstacles. The control inputs of each robot are designed to be bounded. Numerical simulations with different formations are implemented to demonstrate the efficacy of the proposed scheme.
Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learning
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；College of Engineering, Peking University（北京大学工学院）；Sensetime Research（商汤科技研究院）；University of Toronto（多伦多大学）；University College London（伦敦大学学院）；Shanghai AI Lab（上海人工智能实验室）
In this paper, a combination of ankle and hip strategy is used for push recovery of a position-controlled humanoid robot. Ankle strategy and hip strategy are equivalent to Center of Pressure (CoP) and Centroidal Moment Pivot (CMP) regulation respectively. For controlling the CMP and CoP we need a torque-controlled robot, however most of the conventional humanoid robots are position controlled. In this regard, we present an efficient way for implementation of the hip and ankle strategies on a position controlled humanoid robot. We employ a feedback controller to compensate the capture point err…
