This work describes a new human-in-the-loop (HitL) assistive grasping system for individuals with varying levels of physical capabilities. We investigated the feasibility of using four potential input devices with our assistive grasping system interface, using able-bodied individuals to define a set of quantitative metrics that could be used to assess an assistive grasping system. We then took these measurements and created a generalized benchmark for evaluating the effectiveness of any arbitrary input device into a HitL grasping system. The four input devices were a mouse, a speech recognition device, an assistive switch, and a novel sEMG device developed by our group that was connected either to the forearm or behind the ear of the subject. These preliminary results provide insight into how different interface devices perform for generalized assistive grasping tasks and also highlight the potential of sEMG based control for severely disabled individuals.
Universität Hamburg（汉堡大学）；Agile Robots AG（敏捷机器人公司）
Precise robotic grasping of several novel objects is a huge challenge in manufacturing, automation, and logistics. Most of the current methods for model-free grasping are disadvantaged by the sparse data in grasping datasets and by errors in sensor data and contact models. This study combines data generation and sim-to-real transfer learning in a grasping framework that reduces the sim-to-real gap and enables precise and reliable model-free grasping. A large-scale robotic grasping dataset with dense grasp labels is generated using domain randomization methods and a novel data augmentation meth…
UltraDexGrasp: Learning Universal Dexterous Grasping for Bimanual Robots with Synthetic Data
Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）；The University of Hong Kong（香港大学）；Peking University（北京大学）
Social robots, also known as service or assistant robots, have been developed to improve the quality of human life in recent years. Socially assistive robots (SAR) are a special type of social robots that focus on providing support through social interaction. The design of socially capable and intelligent robots can vary, depending on the target user groups. In this work, I assess the effect of socially assistive robots' roles, functions, and communication approaches in the context of a social agent providing service or companionship to users with developmental disabilities. In this thesis, I …
Softbank Robotics Europe（软银机器人欧洲公司）；CRF La Chataigneraie（CRF拉沙泰涅赖机构）
The aim of the present study involves designing a humanoid robot guide as a walking trainer for elderly and rehabilitation patients. The system is based on the humanoid robot Pepper with a compliance approach that allows to match the motion intention of the user to the robot's pace. This feasibility study is backed up by an experimental evaluation conducted in a rehabilitation centre. We hypothesize that Pepper robot used as an assistive partner, can also benefit elderly users by motivating them to perform physical activity.
Carnegie Mellon University（卡内基梅隆大学）；University of Washington（华盛顿大学）
We present the Human And Robot Multimodal Observations of Natural Interactive Collaboration (HARMONIC) data set. This is a large multimodal data set of human interactions with a robotic arm in a shared autonomy setting designed to imitate assistive eating. The data set provides human, robot, and environmental data views of twenty-four different people engaged in an assistive eating task with a 6 degree-of-freedom (DOF) robot arm. From each participant, we recorded video of both eyes, egocentric video from a head-mounted camera, joystick commands, electromyography from the forearm used to opera…
Versatile robotic caregivers could benefit millions of people worldwide, including older adults and people with disabilities. Recent work has explored how robotic caregivers can learn to interact with people through physics simulations, yet transferring what has been learned to real robots remains challenging. Virtual reality (VR) has the potential to help bridge the gap between simulations and the real world. We present Assistive VR Gym (AVR Gym), which enables real people to interact with virtual assistive robots. We also provide evidence that AVR Gym can help researchers improve the perform…
Carnegie Mellon University（卡内基梅隆大学）；Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等理工学院）；Madeira Interactive Technology Institute, University of Madeira（马德拉大学马德拉交互技术研究所）
The research of a socially assistive robot has a potential to augment and assist physical therapy sessions for patients with neurological and musculoskeletal problems (e.g. stroke). During a physical therapy session, generating personalized feedback is critical to improve patient's engagement. However, prior work on socially assistive robotics for physical therapy has mainly utilized pre-defined corrective feedback even if patients have various physical and functional abilities. This paper presents an interactive approach of a socially assistive robot that can dynamically select kinematic feat…
Fudan University（复旦大学）；Westlake University（西湖大学）；Australian National University（澳大利亚国立大学）；Westlake Institute for Advanced Study（西湖高等研究院）
Current NLP techniques have been greatly applied in different domains. In this paper, we propose a human-in-the-loop framework for robotic grasping in cluttered scenes, investigating a language interface to the grasping process, which allows the user to intervene by natural language commands. This framework is constructed on a state-of-the-art rasping baseline, where we substitute a scene-graph representation with a text representation of the scene using BERT. Experiments on both simulation and physical robot show that the proposed method outperforms conventional object-agnostic and scene-grap…
We investigate grasping of rigid objects in unilateral robot-assisted minimally invasive surgery (RAMIS) in this paper. We define a human-centered transparency that quantifies natural action and perception in RAMIS. We demonstrate this human-centered transparency analysis for different values of gripper scaling - the scaling between the grasp aperture of the surgeon-side manipulator and the aperture of the surgical instrument grasper. Thirty-one participants performed teleoperated grasping and perceptual assessment of rigid objects in one of three gripper scaling conditions (fine, normal, and …
Model Reconciliation through Explainability and Collaborative Recovery in Assistive Robotics
German Aerospace Center (DLR), Robotics and Mechatronics Center (RMC)（德国航空航天中心（DLR）、机器人与机电中心）；University of Cambridge, Department of Engineering（剑桥大学工程系）；Universität Tübingen, Department of Psychology（图宾根大学心理学系）；Leibniz-Institut für Wissensmedien (IWM)（莱比锡知识媒体研究所）；Technical University of Munich（慕尼黑技术大学）
MindEye-OmniAssist: A Gaze-Driven LLM-Enhanced Assistive Robot System for Implicit Intention Recognition and Task Execution
Hubei Key Laboratory of Brain-Inspired Intelligent Systems, Huazhong University of Science and Technology（脑启发智能系统湖北省重点实验室，华中科技大学）；Key Laboratory of the Ministry of Education for Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（图像处理与智能控制教育部重点实验室，人工智能与自动化学院，华中科技大学）；state key laboratory of intelligent vehicle safety technology, chongqing changan automobile co ltd（智能车辆安全技术 state key laboratory，重庆长安汽车有限公司）；Science and technology innovation center, China ship development and design centre（科技创新中心，中国船舶工业设计研究中心）
We explore new aspects of assistive living on smart human-robot interaction (HRI) that involve automatic recognition and online validation of speech and gestures in a natural interface, providing social features for HRI. We introduce a whole framework and resources of a real-life scenario for elderly subjects supported by an assistive bathing robot, addressing health and hygiene care issues. We contribute a new dataset and a suite of tools used for data acquisition and a state-of-the-art pipeline for multimodal learning within the framework of the I-Support bathing robot, with emphasis on audi…
While there are many examples in which robots provide social assistance, a lack of theory on how the robots should decide how to assist impedes progress in realizing these technologies. To address this deficiency, we propose a pair of computational models to guide a robot as it provides social assistance. The model of social autonomy helps a robot select an appropriate assistance that will help with the task at hand while also maintaining the autonomy of the person being assisted. The model of social alliance describes how a to determine whether the robot and the person being assisted are coop…
With increasing global age and disability assistive robots are becoming more necessary, and brain computer interfaces (BCI) are often proposed as a solution to understanding the intent of a disabled person that needs assistance. Most frameworks for electroencephalography (EEG)-based motor imagery (MI) BCI control rely on the direct control of the robot in Cartesian space. However, for 3-dimensional movement, this requires 6 motor imagery classes, which is a difficult distinction even for more experienced BCI users. In this paper, we present a simulated training and testing framework which redu…
Creating an Intelligent Dementia-Friendly Living Space: A Feasibility Study Integrating Assistive Robotics, Wearable Sensors, and Spatial Technology
University of Minnesota Duluth（明尼苏达大学德卢斯分校）；Pennsylvania State University-University Park（宾夕法尼亚州立大学公园校区）；University of Minnesota（明尼苏达大学）；Kentucky School of Osteopathic Medicine（肯塔基骨医学院）
Heriot-Watt University（赫瑞-瓦特大学）；University of Glasgow（格拉斯哥大学）
We present an ongoing project that aims to develop a social robot to help children cope with painful and distressing medical procedures in a clinical setting. Our approach uses automated planning as a core component for action selection in order to generate plans that include physical, sensory, and social actions for the robot to use when interacting with humans. A key capability of our system is that the robot's behaviour adapts based on the affective state of the child patient. The robot must operate in a challenging physical and social environment where appropriate and safe interaction with…
There is a close connection between health and the quality of one's social life. Strong social bonds are essential for health and wellbeing, but often health conditions can detrimentally affect a person's ability to interact with others. This can become a vicious cycle resulting in further decline in health. For this reason, the social management of health is an important aspect of healthcare. We propose that socially assistive robots (SARs) could help people with health conditions maintain positive social lives by supporting them in social interactions. This paper makes three contributions, a…
University of Genova（热那亚大学）；Japan Advanced Institute of Science and Technology（北陆先端科学技术大学院大学）；Nagoya University（名古屋大学）；Advinia Health Care Limited LTD（Advinia 医疗保健有限公司）；Chubu University（中部大学）；Softbank Robotics Europe SAS（软银机器人欧洲公司）；University of Bedfordshire（贝德福德大学）；Middlesex University Higher Education Corporation（密德萨斯大学高等教育公司）；Örebro University（厄勒布鲁大学）
The nursing literature shows that cultural competence is an important requirement for effective healthcare. We claim that personal assistive robots should likewise be culturally competent, that is, they should be aware of general cultural characteristics and of the different forms they take in different individuals, and take these into account while perceiving, reasoning, and acting. The CARESSES project is an Europe-Japan collaborative effort that aims at designing, developing and evaluating culturally competent assistive robots. These robots will be able to adapt the way they behave, speak a…
Communication is a hallmark of intelligence. In this work, we present MIRROR, an approach to (i) quickly learn human models from human demonstrations, and (ii) use the models for subsequent communication planning in assistive shared-control settings. MIRROR is inspired by social projection theory, which hypothesizes that humans use self-models to understand others. Likewise, MIRROR leverages self-models learned using reinforcement learning to bootstrap human modeling. Experiments with simulated humans show that this approach leads to rapid learning and more robust models compared to existing b…
Robot grasping is an actively studied area in robotics, mainly focusing on the quality of generated grasps for object manipulation. However, despite advancements, these methods do not consider the human-robot collaboration settings where robots and humans will have to grasp the same objects concurrently. Therefore, generating robot grasps compatible with human preferences of simultaneously holding an object becomes necessary to ensure a safe and natural collaboration experience. In this paper, we propose a novel, deep neural network-based method called CoGrasp that generates human-aware robot …
Socially assistive robots (SARs) aim to provide assistance through social interaction. Previous studies contributed to understanding users` perceptions and preferences regarding existing commercially available SARs. Yet, very few studies regarding SARs' appearance used designated SAR designs, and even fewer evaluated isolated visual qualities (VQ). In this work, we aim to assess the effect of isolated VQs systematically. To achieve this, we first conducted market research and deconstructed the VQs attributed to SARs. Then, a reconstruction of body structure, outline, and color scheme was done,…
