To achieve seamless human-robot interactions, robots need to intimately reason about complex interaction dynamics and future human behaviors within their motion planning process. However, there is a disconnect between state-of-the-art neural network-based human behavior models and robot motion planners -- either the behavior models are limited in their consideration of downstream planning or a simplified behavior model is used to ensure tractability of the planning problem. In this work, we present a framework that fuses together the interpretability and flexibility of trajectory optimization (TO) with the predictive power of state-of-the-art human trajectory prediction models. In particular, we leverage gradient information from data-driven prediction models to explicitly reason about human-robot interaction dynamics within a gradient-based TO problem. We demonstrate the efficacy of our approach in a multi-agent scenario whereby a robot is required to safely and efficiently navigate through a crowd of up to ten pedestrians. We compare against a variety of planning methods, and show that by explicitly accounting for interaction dynamics within the planner, our method offers safer and more efficient behaviors, even yielding proactive and nuanced behaviors such as waiting for a pedestrian to pass before moving.
NIT Surat（苏拉特国家理工学院）；Institute of Technology, University of Tartu（塔尔图大学技术学院）
Most commercially available fixed-wing aerial vehicles (FWV) can carry only small, lightweight computing hardware such as Jetson TX2 onboard. Solving non-linear trajectory optimization on these computing resources is computationally challenging even while considering only the kinematic motion model. Most importantly, the computation time increases sharply as the environment becomes more cluttered. In this paper, we take a step towards overcoming this bottleneck and propose a trajectory optimizer that achieves online performance on both conventional laptops/desktops and Jetson TX2 in a typical …
This paper presents a novel control approach to dealing with object slip during robotic manipulative movements. Slip is a major cause of failure in many robotic grasping and manipulation tasks. Existing works increase grip force to avoid/control slip. However, this may not be feasible when (i) the robot cannot increase the gripping force -- the max gripping force is already applied or (ii) increased force damages the grasped object, such as soft fruit. Moreover, the robot fixes the gripping force when it forms a stable grasp on the surface of an object, and changing the gripping force during r…
Exact Smooth Reformulations for Trajectory Optimization Under Signal Temporal Logic Specifications
Division of Robotics, Perception and Learning, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（机器人、感知与学习 division，电气工程与计算机科学学院，皇家理工学院）
Historical Prediction Attention Mechanism based Trajectory Forecasting for Proactive Work Zone Safety in a Digital Twin Environment
The University of Alabama（阿拉巴马大学）；MITRE Corporation（MITRE公司）；University of Nebraska-Lincoln（内布拉斯加大学林肯分校）；FAMU-FSU College of Engineering（佛罗里达大学-佛罗里达州立大学工程学院）
Stanford University（斯坦福大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
A key challenge for autonomous driving is safe trajectory planning in cluttered, urban environments with dynamic obstacles, such as pedestrians, bicyclists, and other vehicles. A reliable prediction of the future environment, including the behavior of dynamic agents, would allow planning algorithms to proactively generate a trajectory in response to a rapidly changing environment. We present a novel framework that predicts the future occupancy state of the local environment surrounding an autonomous agent by learning a motion model from occupancy grid data using a neural network. We take advan…
Spatiotemporal Receding Horizon Control with Proactive Interaction Towards Autonomous Driving in Dense Traffic
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Great Bay University（大湾区大学）；The Hong Kong University of Science and Technology（香港科技大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）
Surrogate safety measures in the form of conflict indicators are indispensable components of the proactive traffic safety toolbox. Conflict indicators can be classified into past-trajectory-based conflicts and predicted-trajectory-based conflicts. While the calculation of the former class of conflicts is deterministic and unambiguous, the latter category is computed using predicted vehicle trajectories and is thus more stochastic. Consequently, the accuracy of prediction-based conflicts is contingent on the accuracy of the utilized trajectory prediction algorithm. Trajectory prediction can be …
A robotic system which approximates the user intention and appropriate complimentary motion is critical for successful human-robot interaction. %While the existing wearable sensors can monitor human movements in real-time, prediction of human movement is a significant challenge due to its highly non-linear motions optimised through the redundancy in the degrees of freedom. Here, we demonstrate robustness of the Gaussian Process (GP) clustered with a stochastic classification technique for trajectory prediction using an object handover scenario. By parametrising real 6D hand movements during hu…
Stanford University（斯坦福大学）；Toyota Research Institute（丰田研究所）
Multi-agent interactions are important to model for forecasting other agents' behaviors and trajectories. At a certain time, to forecast a reasonable future trajectory, each agent needs to pay attention to the interactions with only a small group of most relevant agents instead of unnecessarily paying attention to all the other agents. However, existing attention modeling works ignore that human attention in driving does not change rapidly, and may introduce fluctuating attention across time steps. In this paper, we formulate an attention model for multi-agent interactions based on a total var…
Trajectory prediction is an important task, especially in autonomous driving. The ability to forecast the position of other moving agents can yield to an effective planning, ensuring safety for the autonomous vehicle as well for the observed entities. In this work we propose a data driven approach based on Markov Chains to generate synthetic trajectories, which are useful for training a multiple future trajectory predictor. The advantages are twofold: on the one hand synthetic samples can be used to augment existing datasets and train more effective predictors; on the other hand, it allows to …
Agentic Reward Modeling: Verifying GUI Agent via Progressive Trajectory-Grounded Interaction
MAIS, Institute of Automation, Chinese Academy of Sciences；School of Artificial Intelligence, University of Chinese Academy of Sciences；School of Computer Science \& Technology, Beijing Jiaotong University
SASI: Leveraging Sub-Action Semantics for Robust Early Action Recognition in Human-Robot Interaction
Institute of Industrial Science, The University of Tokyo（东京大学工业科学研究所）；Interfaculty Initiative in Information Studies, Graduate School of Interdisciplinary Information Studies, The University of Tokyo（东京大学跨学际信息研究学院信息研究联合计划）
Trajectory-based Diffusion from One-shot Human Video for Generalizable Manipulation
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Zerith Robotics（智臻机器人）；University of Science and Technology of China（中国科学技术大学）
Netherlands Organisation for Applied Scientific Research（荷兰应用科学研究组织）；Eindhoven University of Technology（埃因霍温理工大学）
The urban environment is amongst the most difficult domains for autonomous vehicles. The vehicle must be able to plan a safe route on challenging road layouts, in the presence of various dynamic traffic participants such as vehicles, cyclists and pedestrians and in various environmental conditions. The challenge remains to have motion planners that are computationally fast and that account for future movements of other road users proactively. This paper describes an computationally efficient sampling-based trajectory planner for safe and comfortable driving in urban environments. The planner i…
Adap-RPF: Adaptive Trajectory Sampling for Robot Person Following in Dynamic Crowded Environments
Robotics and Computer Vision (RCV) Laboratory, Southern University of Science and Technology (SUSTech), Shenzhen, China（机器人与计算机视觉实验室，南方科技大学，深圳，中国）；SUSTech（南方科技大学）
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Zhejiang Normal University（浙江师范大学）
Vision-and-Language Navigation in Continuous Environments (VLN-CE), which links language instructions to perception and control in the real world, is a core capability of embodied robots. Recently, large-scale pretrained foundation models have been leveraged as shared priors for perception, reasoning, and action, enabling zero-shot VLN without task-specific training. However, existing zero-shot VLN methods depend on costly perception and passive scene understanding, collapsing control to point-level choices. As a result, they are expensive to deploy, misaligned in action semantics, and short-s…
Physical interactions are inevitable part of human-robot collaboration tasks and rather than exhibiting simple reactive behaviors to human interactions, collaborative robots need to be endowed with intuitive behaviors. This paper proposes a trajectory advancement approach that facilitates advancement along a reference trajectory by leveraging assistance from helpful interaction wrench present during human-robot collaboration. We validate our approach through experiments in simulation with iCub.
Recent interest towards autonomous navigation and exploration robots for indoor applications has spurred research into indoor Simultaneous Localization and Mapping (SLAM) robot systems. While most of these SLAM systems use Visual and LiDAR sensors in tandem with an odometry sensor, these odometry sensors drift over time. To combat this drift, Visual SLAM systems deploy compute and memory intensive search algorithms to detect `Loop Closures', which make the trajectory estimate globally consistent. To circumvent these resource (compute and memory) intensive algorithms, we present ViWiD, which in…
Human behavior prediction models enable robots to anticipate how humans may react to their actions, and hence are instrumental to devising safe and proactive robot planning algorithms. However, modeling complex interaction dynamics and capturing the possibility of many possible outcomes in such interactive settings is very challenging, which has recently prompted the study of several different approaches. In this work, we provide a self-contained tutorial on a conditional variational autoencoder (CVAE) approach to human behavior prediction which, at its core, can produce a multimodal probabili…
Efficient routing is one of the key challenges of wireless networking for unmanned autonomous vehicles (UAVs) due to dynamically changing channel and network topology characteristics. Various well known mobile-ad-hoc routing protocols, such as AODV, OLSR and B.A.T.M.A.N. have been proposed to allow for proactive and reactive routing decisions. In this paper, we present a novel approach which leverages application layer knowledge derived from mobility control algorithms guiding the behavior of UAVs to fulfill a dedicated task. Thereby a prediction of future trajectories of the UAVs can be integ…
Enhancing Large Vision Model in Street Scene Semantic Understanding through Leveraging Posterior Optimization Trajectory
Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）；Shenzhen Research Institute of Big Data（大数据研究深圳研究所）；Department of Computer Science and Engineering, Southern University of Science and Technology（南方科技大学计算机科学与工程系）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）
