This paper reports on a data-driven, interaction-aware motion prediction approach for pedestrians in environments cluttered with static obstacles. When navigating in such workspaces shared with humans, robots need accurate motion predictions of the surrounding pedestrians. Human navigation behavior is mostly influenced by their surrounding pedestrians and by the static obstacles in their vicinity. In this paper we introduce a new model based on Long-Short Term Memory (LSTM) neural networks, which is able to learn human motion behavior from demonstrated data. To the best of our knowledge, this is the first approach using LSTMs, that incorporates both static obstacles and surrounding pedestrians for trajectory forecasting. As part of the model, we introduce a new way of encoding surrounding pedestrians based on a 1d-grid in polar angle space. We evaluate the benefit of interaction-aware motion prediction and the added value of incorporating static obstacles on both simulation and real-world datasets by comparing with state-of-the-art approaches. The results show, that our new approach outperforms the other approaches while being very computationally efficient and that taking into account static obstacles for motion predictions significantly improves the prediction accuracy, especially in cluttered environments.
Prediction of human motions is key for safe navigation of autonomous robots among humans. In cluttered environments, several motion hypotheses may exist for a pedestrian, due to its interactions with the environment and other pedestrians. Previous works for estimating multiple motion hypotheses require a large number of samples which limits their applicability in real-time motion planning. In this paper, we present a variational learning approach for interaction-aware and multi-modal trajectory prediction based on deep generative neural networks. Our approach can achieve faster convergence and…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Tilburg University（蒂尔堡大学）；University of Maryland at College Park（马里兰大学帕克分校）
We present a Pedestrian Dominance Model (PDM) to identify the dominance characteristics of pedestrians for robot navigation. Through a perception study on a simulated dataset of pedestrians, PDM models the perceived dominance levels of pedestrians with varying motion behaviors corresponding to trajectory, speed, and personal space. At runtime, we use PDM to identify the dominance levels of pedestrians to facilitate socially-aware navigation for the robots. PDM can predict dominance levels from trajectories with ~85% accuracy. Prior studies in psychology literature indicate that when interactin…
University of Science and Technology of China (USTC)（中国科学技术大学）
It is challenging for a mobile robot to navigate through human crowds. Existing approaches usually assume that pedestrians follow a predefined collision avoidance strategy, like social force model (SFM) or optimal reciprocal collision avoidance (ORCA). However, their performances commonly need to be further improved for practical applications, where pedestrians follow multiple different collision avoidance strategies. In this paper, we propose a map-based deep reinforcement learning approach for crowd-aware robot navigation with various pedestrians. We use the sensor map to represent the envir…
Non-prehensile manipulation methods usually use a simple end effector, e.g., a single rod, to manipulate the object. Compared to the grasping method, such an end effector is compact and flexible, and hence it can perform tasks in a constrained workspace; As a trade-off, it has relatively few degrees of freedom (DoFs), resulting in an under-actuation problem with complex constraints for planning and control. This paper proposes a new non-prehensile manipulation method for the task of object retrieval in cluttered environments, using a rod-like pusher. Specifically, a candidate trajectory in a c…
Uncertainty-aware Gaussian Mixture Model for UWB Time Difference of Arrival Localization in Cluttered Environments
University of Toronto（多伦多大学）；University of Toronto Robotics Institute（多伦多大学机器人研究所）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）；Technical University of Munich（慕尼黑工业大学）；Munich Institute for Robotics and Machine Intelligence（慕尼黑机器人与机器智能研究所）
We present EWareNet, a novel intent and affect-aware social robot navigation algorithm among pedestrians. Our approach predicts the trajectory-based pedestrian intent from gait sequence, which is then used for intent-guided navigation taking into account social and proxemic constraints. We propose a transformer-based model that works on commodity RGB-D cameras mounted onto a moving robot. Our intent prediction routine is integrated into a mapless navigation scheme and makes no assumptions about the environment of pedestrian motion. Our navigation scheme consists of a novel obstacle profile rep…
University of Zurich（苏黎世大学）；ETH Zurich（苏黎世联邦理工学院）；Czech Technical University in Prague（捷克布拉格技术大学）
Recently, neural control policies have outperformed existing model-based planning-and-control methods for autonomously navigating quadrotors through cluttered environments in minimum time. However, they are not perception aware, a crucial requirement in vision-based navigation due to the camera's limited field of view and the underactuated nature of a quadrotor. We propose a learning-based system that achieves perception-aware, agile flight in cluttered environments. Our method combines imitation learning with reinforcement learning (RL) by leveraging a privileged learning-by-cheating framewor…
Australian Centre for Field Robotics, The University of Sydney（悉尼大学澳大利亚野外机器人中心）
Social acceptance is a major hurdle for autonomous vehicle technology, central to which is ensuring both passengers and nearby pedestrians feel safe. This idea of `feeling safe' and perceived safety is highly subjective and rooted in human intuition. As such, traditional analytical approaches to autonomous navigation often fail to cater for the social expectations of individuals. Therefore, this paper proposes an approach to capture the complexity of social expectations and integrate this complexity into a 3-layered Contextual Speed Controller. The layers were; the legal road speed limit, the …
Safe Navigation in Unknown and Cluttered Environments via Direction-Aware Convex Free-Region Generation
Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）机器人与自主系统方向）；Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电子与计算机工程系）
CADGrasp: Learning Contact and Collision Aware General Dexterous Grasping in Cluttered Scenes
Center on Frontiers of Computing Studies, School of Computer Science, Peking University（前沿计算研究中心，计算机学院，北京大学）；National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家级重点实验室，计算机学院，北京大学）
Australian Centre for Field Robotics（澳大利亚野外机器人学中心）；University of Sydney（悉尼大学）
Seamlessly operating an autonomous vehicle in a crowded pedestrian environment is a very challenging task. This is because human movement and interactions are very hard to predict in such environments. Recent work has demonstrated that reinforcement learning-based methods have the ability to learn to drive in crowds. However, these methods can have very poor performance due to inaccurate predictions of the pedestrians' future state as human motion prediction has a large variance. To overcome this problem, we propose a new method, SARL-SGAN-KCE, that combines a deep socially aware attentive val…
SWIFT-Nav: Stability-Aware Waypoint-Level TD3 with Fuzzy Arbitration for UAV Navigation in Cluttered Environments
School of Computer Science and Engineering, University of New South Wales（计算机科学与工程学院，新南威尔士大学）；Escuela de Ingeniería, Universidad Central de Chile（工程学院，智利中央大学）
Avoiding unseen or partially occluded vulnerable road users (VRUs) is a major challenge for fully autonomous driving in urban scenes. However, occlusion-aware risk assessment systems have not been widely studied. Here, we propose a pedestrian emergence estimation and occlusion-aware risk assessment system for urban autonomous driving. First, the proposed system utilizes available contextual information, such as visible cars and pedestrians, to estimate pedestrian emergence probabilities in occluded regions. These probabilities are then used in a risk assessment framework, and incorporated into…
Emerging Extrinsic Dexterity in Cluttered Scenes via Dynamics-aware Policy Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Galbot；Peking University（北京大学）；Shanghai Jiao Tong University（上海交通大学）
University of Michigan（密歇根大学）；University of Massachusetts（马萨诸塞大学）
For automated vehicles (AVs) to reliably navigate through crosswalks, they need to understand pedestrians crossing behaviors. Simple and reliable pedestrian behavior models aid in real-time AV control by allowing the AVs to predict future pedestrian behaviors. In this paper, we present a Behavior aware Model Predictive Controller (B-MPC) for AVs that incorporates long-term predictions of pedestrian crossing behavior using a previously developed pedestrian crossing model. The model incorporates pedestrians gap acceptance behavior and utilizes minimal pedestrian information, namely their positio…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
We present a real-time algorithm, SocioSense, for socially-aware navigation of a robot amongst pedestrians. Our approach computes time-varying behaviors of each pedestrian using Bayesian learning and Personality Trait theory. These psychological characteristics are used for long-term path prediction and generating proximic characteristics for each pedestrian. We combine these psychological constraints with social constraints to perform human-aware robot navigation in low- to medium-density crowds. The estimation of time-varying behaviors and pedestrian personalities can improve the performance…
Instituto de Sistemas e Robótica (LARSyS)（系统与机器人研究所（LARSyS））；Instituto Superior Técnico（高等理工学院）；KTH Royal Institute of Technology（KTH皇家理工学院）
This paper addresses the problem of Human-Aware Navigation (HAN), using multi camera sensors to implement a vision-based person tracking system. The main contributions of this paper are as follows: a novel and efficient Deep Learning person detection and a standardization of human-aware constraints. In the first stage of the approach, we propose to cascade the Aggregate Channel Features (ACF) detector with a deep Convolutional Neural Network (CNN) to achieve fast and accurate Pedestrian Detection (PD). Regarding the human awareness (that can be defined as constraints associated with the robot'…
Safe and efficient co-planning of multiple robots in pedestrian participation environments is promising for applications. In this work, a novel multi-robot social-aware efficient cooperative planner that on the basis of off-policy multi-agent reinforcement learning (MARL) under partial dimension-varying observation and imperfect perception conditions is proposed. We adopt temporal-spatial graph (TSG)-based social encoder to better extract the importance of social relation between each robot and the pedestrians in its field of view (FOV). Also, we introduce K-step lookahead reward setting in mu…
Recent work in robotic manipulation focuses on object retrieval in cluttered spaces under occlusion. Nevertheless, the majority of efforts lack an analysis of conditions for the completeness of the approaches or the methods apply only when objects can be removed from the workspace. This work formulates the general, occlusion-aware manipulation task, and focuses on safe object reconstruction in a confined space with in-place rearrangement. It proposes a framework that ensures safety with completeness guarantees. Furthermore, an algorithm, which is an instantiation of this abstract framework for…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Google（谷歌）；University of Maryland at College Park（马里兰大学帕克分校）
We present a real-time algorithm for emotion-aware navigation of a robot among pedestrians. Our approach estimates time-varying emotional behaviors of pedestrians from their faces and trajectories using a combination of Bayesian-inference, CNN-based learning, and the PAD (Pleasure-Arousal-Dominance) model from psychology. These PAD characteristics are used for long-term path prediction and generating proxemic constraints for each pedestrian. We use a multi-channel model to classify pedestrian characteristics into four emotion categories (happy, sad, angry, neutral). In our validation results, …
This paper presents a novel context-based approach for pedestrian motion prediction in crowded, urban intersections, with the additional flexibility of prediction in similar, but new, environments. Previously, Chen et. al. combined Markovian-based and clustering-based approaches to learn motion primitives in a grid-based world and subsequently predict pedestrian trajectories by modeling the transition between learned primitives as a Gaussian Process (GP). This work extends that prior approach by incorporating semantic features from the environment (relative distance to curbside and status of p…
The potential market for modern self-driving cars is enormous, as they are developing remarkably rapidly. At the same time, however, accidents of pedestrian fatalities caused by autonomous driving have been recorded in the case of street crossing. To ensure traffic safety in self-driving environments and respond to vehicle-human interaction challenges such as jaywalking, we propose Level-$k$ Meta Reinforcement Learning (LK-MRL) algorithm. It takes into account the cognitive hierarchy of pedestrian responses and enables self-driving vehicles to adapt to various human behaviors. %which takes int…
Towards Path-Creative Navigation: Robot Navigation through Embodied Interaction
School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；College of Information Science and Technology, Eastern Institute of Technology（东方理工大学信息科学与技术学院）；School of Mechanical and Power Engineering, Zhengzhou university（郑州大学机械与动力工程学院）；University of Waterloo（滑铁卢大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CAS Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）
Object grasping in cluttered scenes is a widely investigated field of robot manipulation. Most of the current works focus on estimating grasp pose from point clouds based on an efficient single-shot grasp detection network. However, due to the lack of geometry awareness of the local grasping area, it may cause severe collisions and unstable grasp configurations. In this paper, we propose a two-stage grasp pose refinement network which detects grasps globally while fine-tuning low-quality grasps and filtering noisy grasps locally. Furthermore, we extend the 6-DoF grasp with an extra dimension a…
University of British Columbia（不列颠哥伦比亚大学）；Monash University（莫纳什大学）
In this paper, we propose a novel navigation system for mobile robots in pedestrian-rich sidewalk environments. Sidewalks are unique in that the pedestrian-shared space has characteristics of both roads and indoor spaces. Like vehicles on roads, pedestrian movement often manifests as linear flows in opposing directions. On the other hand, pedestrians also form crowds and can exhibit much more random movements than vehicles. Classical algorithms are insufficient for safe navigation around pedestrians and remaining on the sidewalk space. Thus, our approach takes advantage of natural human motion…
Geometry-Aware Safety-Critical Local Reactive Controller for Robot Navigation in Unknown and Cluttered Environments
The Hong Kong University of Science and Technology（香港科技大学）；Hong Kong Baptist University（香港浸会大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
TANGO: Humanoid Navigation in Cluttered Environments with a Whole-Body Vision-Language-Action Model
University of California, Berkeley（加州大学伯克利分校）；Peking University（北京大学）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；Princeton University（普林斯顿大学）
OA-ECBVC: A Cooperative Collision-free Encirclement and Capture Approach in Cluttered Environments
The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；Hebei University of Science and Technology（河北科技大学）；Tongji University（同济大学）
NavIsaacLab: Generating Realistic Crowd via Parallel Robot Learning for Benchmarking Human-aware Navigation
Southern University of Science and Technology（南方科技大学）；National University of Singapore（新加坡国立大学）；Peng Cheng Laboratory（鹏城实验室）；Jiaxing Research Institute, Southern University of Science and Technology（南方科技大学嘉兴研究院）
Queen’s University Belfast（贝尔法斯特女王大学）；Northumbria University（诺森比亚大学）；Durham University（杜伦大学）
Motion control algorithms in the presence of pedestrians are critical for the development of safe and reliable Autonomous Vehicles (AVs). Traditional motion control algorithms rely on manually designed decision-making policies which neglect the mutual interactions between AVs and pedestrians. On the other hand, recent advances in Deep Reinforcement Learning allow for the automatic learning of policies without manual designs. To tackle the problem of decision-making in the presence of pedestrians, the authors introduce a framework based on Social Value Orientation and Deep Reinforcement Learnin…
Solving a collision-aware multi-agent mission planning (task allocation and path finding) problem is challenging due to the requirement of real-time computational performance, scalability, and capability of handling static/dynamic obstacles and tasks in a cluttered environment. This paper proposes a distributed real-time (on the order of millisecond) algorithm DrMaMP, which partitions the entire unassigned task set into subsets via approximation and decomposes the original problem into several single-agent mission planning problems. This paper presents experiments with dynamic obstacles and ta…
Knowledge-aware Graph Transformer for Pedestrian Trajectory Prediction
Southern University of Science and Technology (SUSTech)（南方科技大学）；City University of Hong Kong（香港城市大学）；Guangdong Polytechnic Normal University（广东技术师范大学）；The University of Sydney（悉尼大学）；The University of Adelaide（阿德莱德大学）
