As robotic technology advances, the barriers to the coexistence of humans and robots are slowly coming down. Application domains like elderly care, collaborative manufacturing, collaborative manipulation, etc., are considered the need of the hour, and progress in robotics holds the potential to address many societal challenges. The future socio-technical systems constitute of blended workforce with a symbiotic relationship between human and robot partners working collaboratively. This thesis attempts to address some of the research challenges in enabling human-robot collaboration. In particular, the challenge of a holistic perception of a human partner to continuously communicate his intentions and needs in real-time to a robot partner is crucial for the successful realization of a collaborative task. Towards that end, we present a holistic human perception framework for real-time monitoring of whole-body human motion and dynamics. On the other hand, the challenge of leveraging assistance from a human partner will lead to improved human-robot collaboration. In this direction, we attempt at methodically defining what constitutes assistance from a human partner and propose partner-aware robot control strategies to endow robots with the capacity to meaningfully engage in a collaborative task.
Perception of Social Robots as Communication Partners in Healthcare for Older Adults
Institute for Anthropomatics and Robotics（人机学与机器人研究所）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Department of Robotics（机器人系）；Tohoku University（东北大学）；Institute of Medical Psychology（医学心理学研究所）；Heidelberg University（海德堡大学）；Institute of Psychology（心理学研究所）
Catch, Throw, Repeat: Planning for Human-Robot Partner Juggling
Technical University of Darmstadt（达姆施塔特工业大学）；Justus-Liebig University Gießen（吉森尤斯 - 李比希大学）；German Research Center for AI (DFKI)（德国人工智能研究中心（DFKI））；Robotics Institute Germany（德国机器人研究所）；Indian Institute of Technology Delhi - Abu Dhabi（印度理工学院德里分校 - 阿布扎比）
We present a reinforcement learning based framework for human-centered collaborative systems. The framework is proactive and balances the benefits of timely actions with the risk of taking improper actions by minimizing the total time spent to complete the task. The framework is learned end-to-end in an unsupervised fashion addressing the perception uncertainties and decision making in an integrated manner. The framework is shown to provide more fluent coordination between human and robot partners on an example task of packaging compared to alternatives for which perception and decision-making…
Current technological advances open up new opportunities for bringing human-machine interaction to a new level of human-centered cooperation. In this context, a key issue is the semantic understanding of the environment in order to enable mobile robots more complex interactions and a facilitated communication with humans. Prerequisites are the vision-based registration of semantic objects and humans, where the latter are further analyzed for potential interaction partners. Despite significant research achievements, the reliable and fast registration of semantic information still remains a chal…
How human-like do conversational robots need to look to enable long-term human-robot conversation? One essential aspect of long-term interaction is a human's ability to adapt to the varying degrees of a conversational partner's engagement and emotions. Prosodically, this can be achieved through (dis)entrainment. While speech-synthesis has been a limiting factor for many years, restrictions in this regard are increasingly mitigated. These advancements now emphasise the importance of studying the effect of robot embodiment on human entrainment. In this study, we conducted a between-subjects onli…
A holistic perception system of internal and external monitoring for ground autonomous vehicles: AutoTRUST paradigm
Industrial Systems Institute, ATHENA Research Center, Patras Science Park, Greece（工业系统研究所，ATHENA研究中心，帕特拉斯科学公园，希腊）；AviSense.AI, Patras Science Park, Greece（AviSense.AI，帕特拉斯科学公园，希腊）；Information Technologies Institute, CERTH, Greece（信息科技研究所，CERTH，希腊）；KIOS Research and Innovation Center of Excellence, University of Cyprus, Cyprus（KIOS研究与创新中心，塞浦路斯大学，塞浦路斯）
University of Hertfordshire（赫特福德大学）；University of Waterloo（滑铁卢大学）
A challenge in using robots in human-inhabited environments is to design behavior that is engaging, yet robust to the perturbations induced by human interaction. Our idea is to imbue the robot with intrinsic motivation (IM) so that it can handle new situations and appears as a genuine social other to humans and thus be of more interest to a human interaction partner. Human-robot interaction (HRI) experiments mainly focus on scripted or teleoperated robots, that mimic characteristics such as IM to control isolated behavior factors. This article presents a "robotologist" study design that allows…
Jožef Stefan Institute（约泽夫·斯蒂芬研究所）；Jožef Stefan International Postgraduate School（约泽夫·斯蒂芬国际研究生院）
An important factor in developing control models for human-robot collaboration is how acceptable they are to their human partners. One such method for creating acceptable control models is to attempt to mimic human-like behaviour in robots so that their actions appear more intuitive to humans. To investigate how task complexity affects human perception and acceptance of their robot partner, we propose a novel human-based robot control model for obstacle avoidance that can account for the leader-follower dynamics that normally occur in human collaboration. The performance and acceptance of the …
EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI
The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）
Austrian Research Institute for Artificial Intelligence（奥地利人工智能研究所）；Joanneum Research（乔安娜姆研究院）；Johannes Kepler University（约翰内斯·开普勒大学）
With increased applications of collaborative robots (cobots) in industrial workplaces, behavioural effects of human-cobot interactions need to be further investigated. This is of particular importance as nonverbal behaviours of collaboration partners in human-robot teams significantly influence the experience of the human interaction partners and the success of the collaborative task. During the Ars Electronica 2020 Festival for Art, Technology and Society (Linz, Austria), we invited visitors to exploratively interact with an industrial robot, exhibiting restricted interaction capabilities: ex…
The Chinese University of Hong Kong（香港中文大学）；T stone Robotics Institute（T石机器人研究所）；Department of Computer Science and Engineering（计算机科学与工程学系）
Laparoscopic Field of View (FOV) control is one of the most fundamental and important components in Minimally Invasive Surgery (MIS), nevertheless, the traditional manual holding paradigm may easily bring fatigue to surgical assistants, and misunderstanding between surgeons also hinders assistants to provide a high-quality FOV. Targeting this problem, we here present a data-driven framework to realize an automated laparoscopic optimal FOV control. To achieve this goal, we offline learn a motion strategy of laparoscope relative to the surgeon's hand-held surgical tool from our in-house surgical…
Carnegie Mellon University（卡内基梅隆大学）；Meta Reality Lab（Meta现实实验室）；University of Toronto（多伦多大学）
Understanding and modeling human behavior is fundamental to almost any computer vision and robotics applications that involve humans. In this thesis, we take a holistic approach to human behavior modeling and tackle its three essential aspects -- simulation, perception, and generation. Throughout the thesis, we show how the three aspects are deeply connected and how utilizing and improving one aspect can greatly benefit the other aspects. We also discuss the lessons learned and our vision for what is next for human behavior modeling.
From Prompt to Physical Actuation: Holistic Threat Modeling of LLM-Enabled Robotic Systems
School of Informatics, Computing, and Cyber Systems, Northern Arizona University（信息学、计算与网络系统学院，北亚利桑那大学）；Department of Software Science, Tallinn University of Technology（软件科学系，塔林技术大学）
New York University（纽约大学）；Shanghai Jiao Tong University（上海交通大学）；Rutgers University（罗格斯大学）
To promote better performance-bandwidth trade-off for multi-agent perception, we propose a novel distilled collaboration graph (DiscoGraph) to model trainable, pose-aware, and adaptive collaboration among agents. Our key novelties lie in two aspects. First, we propose a teacher-student framework to train DiscoGraph via knowledge distillation. The teacher model employs an early collaboration with holistic-view inputs; the student model is based on intermediate collaboration with single-view inputs. Our framework trains DiscoGraph by constraining post-collaboration feature maps in the student mo…
Istituto Italiano di Tecnologia（意大利理工学院）；University of Naples Federico II（那不勒斯费德里科二世大学）
Enabling robots to work in close proximity to humans necessitates a control framework that does not only incorporate multi-sensory information for autonomous and coordinated interactions but also has perceptive task planning to ensure an adaptable and flexible collaborative behaviour. In this research, an intuitive stack-of-tasks (iSoT) formulation is proposed, that defines the robot's actions by considering the human-arm postures and the task progression. The framework is augmented with visuo-tactile information to effectively perceive the collaborative environment and intuitively switch betw…
EHC-MM: Embodied Holistic Control for Mobile Manipulation
Beijing Samsung Telecom R&D Center（北京三星通信技术研发中心）；National Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；PKU-WUHAN Institute for Artificial Intelligence（北京大学-武汉人工智能研究院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Tsinghua University（清华大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Everyday Robots（每日机器人公司）；Georgia Institute of Technology（佐治亚理工学院）；Robotics at Google（谷歌机器人部门）；Stanford University（斯坦福大学）
As robots become more prevalent, optimizing their design for better performance and efficiency is becoming increasingly important. However, current robot design practices overlook the impact of perception and design choices on a robot's learning capabilities. To address this gap, we propose a comprehensive methodology that accounts for the interplay between the robot's perception, hardware characteristics, and task requirements. Our approach optimizes the robot's morphology holistically, leading to improved learning and task execution proficiency. To achieve this, we introduce a Morphology-AGn…
Multi-modal perception for soft robotic interactions using generative models
The BioRobotics Institute, Sant’Anna School of Advanced Studies（圣安娜高级研究院生物机器人研究所）；Sant’Anna School of Advanced Studies（圣安娜高级研究院）；University College London（伦敦大学学院）
Assessing Vision-Language Models for Perception in Autonomous Underwater Robotic Software
Simula Research Laboratory（Simula研究实验室）；Oslo Metropolitan University（奥斯陆城市大学）；Mondragon University（蒙德拉贡大学）；National Institute of Informatics（国立信息学研究所）；Det norske Veritas (DNV)（挪威船级社（DNV））
Collaborative Trajectory Prediction via Late Fusion
Khalifa University, Computer Science（卡利法大学，计算机科学）；New York University Abu Dhabi, Computer Science（纽约大学阿布扎赫德分校，计算机科学）；Khalifa University, Computer and Information Engineering（卡利法大学，计算机与信息工程）；Khalifa University, KUCARS-KU Center for Autonomous Robotic Systems, Department of Computer Science（卡利法大学，KUCARS-KU自主机器人系统中心，计算机科学系）；Skolkovo Institute of Science, Intelligent Space Robotics Laboratory, Center for Engineering Systems and Sciences, and Technology（斯科尔科沃科学研究所，智能空间机器人实验室，工程系统与科学中心，和技术）
Breaking the Latency Barrier: Synergistic Perception and Control for High-Frequency 3D Ultrasound Servoing
Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China（香港中文大学电子工程系）；Department of Biomedical Engineering, Tsinghua University, Beijing, China（清华大学生物医学工程系）；Great Bay University, Dongguan, China（东莞大邦大学）
