Robot awareness of human actions is an essential research problem in robotics with many important real-world applications, including human-robot collaboration and teaming. Over the past few years, depth sensors have become a standard device widely used by intelligent robots for 3D perception, which can also offer human skeletal data in 3D space. Several methods based on skeletal data were designed to enable robot awareness of human actions with satisfactory accuracy. However, previous methods treated all body parts and features equally important, without the capability to identify discriminative body parts and features. In this paper, we propose a novel simultaneous Feature And Body-part Learning (FABL) approach that simultaneously identifies discriminative body parts and features, and efficiently integrates all available information together to enable real-time robot awareness of human behaviors. We formulate FABL as a regression-like optimization problem with structured sparsity-inducing norms to model interrelationships of body parts and features. We also develop an optimization algorithm to solve the formulated problem, which possesses a theoretical guarantee to find the optimal solution. To evaluate FABL, three experiments were performed using public benchmark datasets, including the MSR Action3D and CAD-60 datasets, as well as a Baxter robot in practical assistive living applications. Experimental results show that our FABL approach obtains a high recognition accuracy with a processing speed of the order-of-magnitude of 10e4 Hz, which makes FABL a promising method to enable real-time robot awareness of human behaviors in practical robotics applications.
Alternative Interfaces for Human-initiated Natural Language Communication and Robot-initiated Haptic Feedback: Towards Better Situational Awareness in Human-Robot Collaboration
If They Disagree, Will You Conform? Exploring the Role of Robots' Value Awareness in a Decision-Making Task
CONTACT Unit, Italian Institute of Technology, Genoa, Italy（意大利理工学院CONTACT单元，热那亚，意大利）；IDLab-AIRO, Ghent University – imec, Belgium（根特大学imec IDLab-AIRO，比利时）
A Replicable Robotics Awareness Method Using LLM-Enabled Robotics Interaction: Evidence from a Corporate Challenge
KINESIS Core Technology Platform, New York University Abu Dhabi (NYUAD)（纽约大学阿布扎比分校KINESIS核心科技平台）；S.M.A.R.T. Construction Research Group, Division of Engineering, New York University Abu Dhabi (NYUAD)（纽约大学阿布扎比分校S.M.A.R.T.建筑研究小组）；AD Ports Group, Corporate Innovation Department（AD Ports集团企业创新部）
An Exploratory Study on Human-Robot Interaction using Semantics-based Situational Awareness
Extreme Robotics Lab (ERL) and National Center for Nuclear Robotics (NCNR), University of Birmingham, UK（极端机器人实验室（ERL）和核机器人国家中心（NCNR）、英国伯明翰大学）；School of Electronic Engineering and Computer Science, Queen Mary University of London, UK（电子工程与计算机科学学院，伦敦女王玛丽大学）
We consider the design space available for social robots in terms of a hierarchy of functional definitions: the essential properties in terms of a locus of interaction, autonomy, intelligence, awareness of humans as possessors of mental state, and awareness of humans as social interactors. We also suggest that the emphasis on physical embodiment in some segments of the social robotics community has obscured commonalities with a class of agents that are identical in all other respects. These definitions naturally suggest research issues, directions, and possibilities which we explore. Social ro…
This paper introduces the notion of danger awareness in the context of Human-Robot Interaction (HRI), which decodes whether a human is aware of the existence of the robot, and illuminates whether the human is willing to engage in enforcing the safety. This paper also proposes a method to quantify this notion as a single binary variable, so-called danger awareness coefficient. By analyzing the effect of this coefficient on the human's actions, an online Bayesian learning method is proposed to update the belief about the value of the coefficient. It is shown that based upon the danger awareness …
University of Michigan（密歇根大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）
As social robots become increasingly prevalent in day-to-day environments, they will participate in conversations and appropriately manage the information shared with them. However, little is known about how robots might appropriately discern the sensitivity of information, which has major implications for human-robot trust. As a first step to address a part of this issue, we designed a privacy controller, CONFIDANT, for conversational social robots, capable of using contextual metadata (e.g., sentiment, relationships, topic) from conversations to model privacy boundaries. Afterwards, we condu…
Detecting other agents and forecasting their behavior is an integral part of the modern robotic autonomy stack, especially in safety-critical scenarios entailing human-robot interaction such as autonomous driving. Due to the importance of these components, there has been a significant amount of interest and research in perception and trajectory forecasting, resulting in a wide variety of approaches. Common to most works, however, is the use of the same few accuracy-based evaluation metrics, e.g., intersection-over-union, displacement error, log-likelihood, etc. While these metrics are informat…
This paper presents an experimentation platform for human robot collaboration as a system of systems as well as proposes a conceptual framework describing the aspects of Human Robot Collaboration. These aspects are Awareness, Intelligence and Compliance of the system. Based on this framework case studies describing experiment setups performed using this platform are discussed. Each experiment highlights the use of the subsystems such as the digital twin, motion capture system, human-physiological monitoring system, data collection system and robot control and interface systems. A highlight of …
The article introduces the concept of "diversity-aware" robotics and discusses the need to develop computational models to embed robots with diversity-awareness: that is, robots capable of adapting and re-configuring their behavior to recognize, respect, and value the uniqueness of the person they interact with to promote inclusion regardless of their age, race, gender, cognitive or physical capabilities, etc. Finally, the article discusses possible technical solutions based on Ontologies and Bayesian Networks, starting from previous experience with culturally competent robots.
Fluent human-human teaming is often characterized by tacit interaction without explicit communication. This is because explicit communication, such as language utterances and gestures, are inherently interruptive. On the other hand, tacit interaction requires team situation awareness (TSA) to facilitate, which often relies on explicit communication to maintain, creating a paradox. In this paper, we consider implicit and naturalistic team status projection for tacit human-robot interaction. Implicitness minimizes interruption while naturalness reduces cognitive demand, and they together improve…
Human-machine interaction (HMI) and human-robot interaction (HRI) can assist structural monitoring and structural dynamics testing in the laboratory and field. In vibratory experimentation, one mode of generating vibration is to use electrodynamic exciters. Manual control is a common way of setting the input of the exciter by the operator. To measure the structural responses to these generated vibrations sensors are attached to the structure. These sensors can be deployed by repeatable robots with high endurance, which require on-the-fly control. If the interface between operators and the cont…
Institut de Robòtica i Informàtica Industrial (CSIC-UPC)（工业机器人与信息研究所（西班牙国家研究委员会-加泰罗尼亚理工大学））
In this work we argue that in Human-Robot Collaboration (HRC) tasks, the Perception-Action cycle in HRC tasks can not fully explain the collaborative behaviour of the human and robot and it has to be extended to Perception-Intention-Action cycle, where Intention is a key topic. In some cases, agent Intention can be perceived or inferred by the other agent, but in others, it has to be explicitly informed to the other agent to succeed the goal of the HRC task. The Perception-Intention-Action cycle includes three basic functional procedures: Perception-Intention, Situation Awareness and Action. T…
Training visual control policies from scratch on a new robot typically requires generating large amounts of robot-specific data. How might we leverage data previously collected on another robot to reduce or even completely remove this need for robot-specific data? We propose a "robot-aware control" paradigm that achieves this by exploiting readily available knowledge about the robot. We then instantiate this in a robot-aware model-based RL policy by training modular dynamics models that couple a transferable, robot-aware world dynamics module with a robot-specific, potentially analytical, robo…
ILeSiA: Interactive Learning of Robot Situational Awareness from Camera Input
Czech Institute of Informatics, Robotics, and Cybernetics, Czech Technical University in Prague, Czech Republic（捷克信息学、机器人学与自动控制研究所，布拉格捷克技术大学）；Cognitive Robotics, Delft University of Technology, The Netherlands（认知机器人学，代尔夫特理工大学）
Vision-Based Human Awareness Estimation for Enhanced Safety and Efficiency of AMRs in Industrial Warehouses
Fraunhofer Austria Research GmbH（弗劳恩霍夫奥地利研究有限公司）；Computer Vision Lab（计算机视觉实验室）；Digital Factory Vorarlberg GmbH（数字工厂沃尔夫斯堡有限公司）；Institute of Computer Technology（计算机技术研究所）
During teleoperation of a mobile robot, providing good operator situation awareness is a major concern as a single mistake can lead to mission failure. Camera streams are widely used for teleoperation but offer limited field-of-view. In this paper, we present a flexible framework for virtual projections to increase situation awareness based on a novel method to fuse multiple cameras mounted anywhere on the robot. Moreover, we propose a complementary approach to improve scene understanding by fusing camera images and geometric 3D Lidar data to obtain a colorized point cloud. The implementation …
Collaborative multi-robot perception provides multiple views of an environment, offering varying perspectives to collaboratively understand the environment even when individual robots have poor points of view or when occlusions are caused by obstacles. These multiple observations must be intelligently fused for accurate recognition, and relevant observations need to be selected in order to allow unnecessary robots to continue on to observe other targets. This research problem has not been well studied in the literature yet. In this paper, we propose a novel approach to collaborative multi-robo…
Designing robotic assistance devices for manipulation tasks is challenging. This work is concerned with improving accuracy and usability of semi-autonomous robots, such as human operated manipulators or exoskeletons. The key insight is to develop a system that takes into account context- and user-awareness to take better decisions in how to assist the user. The context-awareness is implemented by enabling the system to automatically generate a set of candidate grasps and reach-to-grasp trajectories in novel, cluttered scenes. The user-awareness is implemented as a linear time-variant feedback …
EqvAfford: SE(3) Equivariance for Point-Level Affordance Learning
School of EECS, PKU（北京大学工程与计算机科学学院）；CFCS, School of CS, PKU（北京大学计算机学院计算与认知科学中心）；National Key Laboratory for Multimedia Information Processing, School of CS, PKU（北京大学计算机学院多媒体信息处理国家重点实验室）
We present the first fully distributed multi-robot system for dense metric-semantic Simultaneous Localization and Mapping (SLAM). Our system, dubbed Kimera-Multi, is implemented by a team of robots equipped with visual-inertial sensors, and builds a 3D mesh model of the environment in real-time, where each face of the mesh is annotated with a semantic label (e.g., building, road, objects). In Kimera-Multi, each robot builds a local trajectory estimate and a local mesh using Kimera. Then, when two robots are within communication range, they initiate a distributed place recognition and robust po…
Oxford Brookes University（牛津布鲁克斯大学）；University of Naples Federico II（那不勒斯费德里科二世大学）；ETH Zurich（苏黎世联邦理工学院）；University of Science and Technology of Mazandaran（马赞达兰科技大学）；Siemens SpA（西门子股份公司（意大利））
Humans drive in a holistic fashion which entails, in particular, understanding dynamic road events and their evolution. Injecting these capabilities in autonomous vehicles can thus take situational awareness and decision making closer to human-level performance. To this purpose, we introduce the ROad event Awareness Dataset (ROAD) for Autonomous Driving, to our knowledge the first of its kind. ROAD is designed to test an autonomous vehicle's ability to detect road events, defined as triplets composed by an active agent, the action(s) it performs and the corresponding scene locations. ROAD comp…
