Recent work in the construction of 3D scene graphs has enabled mobile robots to build large-scale metric-semantic hierarchical representations of the world. These detailed models contain information that is useful for planning, however an open question is how to derive a planning domain from a 3D scene graph that enables efficient computation of executable plans. In this work, we present a novel approach for defining and solving Task and Motion Planning problems in large-scale environments using hierarchical 3D scene graphs. We describe a method for building sparse problem instances which enables scaling planning to large scenes, and we propose a technique for incrementally adding objects to that domain during planning time that minimizes computation on irrelevant elements of the scene graph. We evaluate our approach in two real scene graphs built from perception, including one constructed from the KITTI dataset. Furthermore, we demonstrate our approach in the real world, building our representation, planning in it, and executing those plans on a real robotic mobile manipulator. A video supplement is available at \url{https://youtu.be/v8fkwLjBn58}.
Texas A&M University（德克萨斯农工大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Instituto Tecnológico Autónomo de México (ITAM)（墨西哥自治技术研究所）
We present a hierarchical skeleton-guided motion planning algorithm to guide mobile robots. A good skeleton maps the connectivity of the subspace of c-space containing significant degrees of freedom and is able to guide the planner to find the desired solutions fast. However, sometimes the skeleton does not closely represent the free c-space, which often misleads current skeleton-guided planners. The hierarchical skeleton-guided planning strategy gradually relaxes its reliance on the workspace skeleton as C space is sampled, thereby incrementally returning a sub-optimal path, a feature that is…
Situationally-aware Path Planning Exploiting 3D Scene Graphs
Automation and Robotics Research Group, Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg（自动化与机器人研究组，安全、可靠与信任跨学科研究中心，卢森堡大学）；Faculty of Science, Technology and Medicine, University of Luxembourg（科学、技术与医学学院，卢森堡大学）
College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）；Huzhou Institute, Zhejiang University（浙江大学湖州研究院）
In this paper, we describe a robust multi-drone planning framework for high-speed trajectories in large scenes. It uses a free-space-oriented map to free the optimization from cumbersome environment data. A capsule-like safety constraint is designed to avoid reciprocal collisions when vehicles deviate from their nominal flight progress under disturbance. We further show the minimum-singularity differential flatness of our drone dynamics with nonlinear drag effects involved. Leveraging the flatness map, trajectory optimization is efficiently conducted on the flat outputs while still subject to …
Search, Ground, Plan: Functional Sufficiency for Task and Motion Planning under Incomplete Scene Knowledge
Robotics Research Center, IIIT Hyderabad（海得拉巴国际信息技术学院机器人研究中心）；Center for Security, Theory and Algorithmic Research, IIIT Hyderabad（海得拉巴国际信息技术学院安全、理论与算法研究中心）
University of Hong Kong（香港大学）；Harbin Institute of Technology（哈尔滨工业大学）
Recent advances in LiDAR technology have opened up new possibilities for robotic navigation. Given the widespread use of occupancy grid maps (OGMs) in robotic motion planning, this paper aims to address the challenges of integrating LiDAR with OGMs. To this end, we propose ROG-Map, a uniform grid-based OGM that maintains a local map moving along with the robot to enable efficient map operation and reduce memory costs for large-scene autonomous flight. Moreover, we present a novel incremental obstacle inflation method that significantly reduces the computational cost of inflation. The proposed …
Institute of Cyber-Systems and Control, Zhejiang University（浙江大学控制科学与工程学院 cyber系统研究所）；Polytechnic Institute of Zhejiang University（浙江大学理工学院）；Shanghai AI Laboratory（上海人工智能实验室）
OceanPlan: Hierarchical Planning and Replanning for Natural Language AUV Piloting in Large-scale Unexplored Ocean Environments
School of Electrical and Computer Engineering, Georgia Institute of Technology（佐治亚理工学院电气与计算机工程学院）；The Hong Kong University of Science and Technology（香港科技大学）；School of Electrical Engineering, University of Notre Dame（圣母大学电气工程学院）
Applied AI Lab (A2I)（应用人工智能实验室（A2I））；Dynamic Robot Systems (DRS)（动态机器人系统实验室（DRS））；Oxford Robotics Institute (ORI)（牛津机器人研究所（ORI））；University of Oxford（牛津大学）
Motion planning framed as optimisation in structured latent spaces has recently emerged as competitive with traditional methods in terms of planning success while significantly outperforming them in terms of computational speed. However, the real-world applicability of recent work in this domain remains limited by the need to express obstacle information directly in state-space, involving simple geometric primitives. In this work we address this challenge by leveraging learned scene embeddings together with a generative model of the robot manipulator to drive the optimisation process. In addit…
University of Hamburg（汉堡大学）；TU Darmstadt（达姆施塔特工业大学）；Tampere University（坦佩雷大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
3D scene models are useful in robotics for tasks such as path planning, object manipulation, and structural inspection. We consider the problem of creating a 3D model using depth images captured by a team of multiple robots. Each robot selects a viewpoint and captures a depth image from it, and the images are fused to update the scene model. The process is repeated until a scene model of desired quality is obtained. Next-best-view planning uses the current scene model to select the next viewpoints. The objective is to select viewpoints so that the images captured using them improve the quality…
In order to solve complex, long-horizon tasks, intelligent robots need to carry out high-level, abstract planning and reasoning in conjunction with motion planning. However, abstract models are typically lossy and plans or policies computed using them can be inexecutable. These problems are exacerbated in stochastic situations where the robot needs to reason about and plan for multiple contingencies. We present a new approach for integrated task and motion planning in stochastic settings. In contrast to prior work in this direction, we show that our approach can effectively compute integrated …
Bidirectional Task-Motion Planning Based on Hierarchical Reinforcement Learning for Strategic Confrontation
School of Automation Science and Electrical Engineering, Beihang University（自动化科学与电气工程学院，北京航空航天大学）；Advanced Research Institute of Multidisciplinary Sciences and State Key Laboratory of CNS/ATM, Beijing Institute of Technology（多学科科学先进研究院和 CNS/ATM 国家重点实验室，北京理工大学）
ETH Zürich（苏黎世联邦理工学院）；Technical University of Munich（慕尼黑工业大学）；Google（谷歌）
Exploration of unknown environments is a fundamental problem in robotics and an essential component in numerous applications of autonomous systems. A major challenge in exploring unknown environments is that the robot has to plan with the limited information available at each time step. While most current approaches rely on heuristics and assumption to plan paths based on these partial observations, we instead propose a novel way to integrate deep learning into exploration by leveraging 3D scene completion for informed, safe, and interpretable exploration mapping and planning. Our approach, SC…
University of Stuttgart（斯图加特大学）；Max-Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；TU Berlin（柏林工业大学）
In this paper, we propose a deep convolutional recurrent neural network that predicts action sequences for task and motion planning (TAMP) from an initial scene image. Typical TAMP problems are formalized by combining reasoning on a symbolic, discrete level (e.g. first-order logic) with continuous motion planning such as nonlinear trajectory optimization. Due to the great combinatorial complexity of possible discrete action sequences, a large number of optimization/motion planning problems have to be solved to find a solution, which limits the scalability of these approaches. To circumvent thi…
Wild-Drive: Off-Road Scene Captioning and Path Planning via Robust Multi-modal Routing and Efficient Large Language Model
School of Instrument Science and Engineering, Southeast University（东南大学仪器科学与工程学院）；Southeast University Nanjing Jiangbei New Area Innovation Research Institute（东南大学南京江滨新区创新研究院）；National Key Laboratory of Equipment State Sensing and Smart Support, National University of Defense Technology（国防科技大学装备状态感知与智能支撑国家重点实验室）；School of Transportation, Shandong University of Science and Technology（山东科技大学交通学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Otto von Guericke University（奥托·冯·格里克大学）；German Aerospace Center (DLR)（德国航空航天中心（DLR））；Technical University of Darmstadt（达姆施塔特工业大学）
Planning and control for autonomous vehicles usually are hierarchical separated. However, increasing performance demands and operating in highly dynamic environments requires an frequent re-evaluation of the planning and tight integration of control and planning to guarantee safety. We propose an integrated hierarchical predictive control and planning approach to tackle this challenge. Planner and controller are based on the repeated solution of moving horizon optimal control problems. The planner can choose different low-layer controller modes for increased flexibility and performance instead…
Learning to Plan, Planning to Learn: Adaptive Hierarchical RL-MPC for Sample-Efficient Decision Making
Toyota Research Institute（丰田研究院）；Learning to Plan, Planning to Learn: Adaptive Hierarchical RL-MPC for Sample-Efficient Decision Making（学习规划，规划学习：自适应分层RL-MPC用于样本高效决策）
Competent multi-lane cruising requires using lane changes and within-lane maneuvers to achieve good speed and maintain safety. This paper proposes a design for autonomous multi-lane cruising by combining a hierarchical reinforcement learning framework with a novel state-action space abstraction. While the proposed solution follows the classical hierarchy of behavior decision, motion planning and control, it introduces a key intermediate abstraction within the motion planner to discretize the state-action space according to high level behavioral decisions. We argue that this design allows princ…
NVIDIA（英伟达）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
We present a visually grounded hierarchical planning algorithm for long-horizon manipulation tasks. Our algorithm offers a joint framework of neuro-symbolic task planning and low-level motion generation conditioned on the specified goal. At the core of our approach is a two-level scene graph representation, namely geometric scene graph and symbolic scene graph. This hierarchical representation serves as a structured, object-centric abstraction of manipulation scenes. Our model uses graph neural networks to process these scene graphs for predicting high-level task plans and low-level motions. W…
Massachusetts Institute of Technology（麻省理工学院）；University of Washington（华盛顿大学）；NVIDIA（英伟达）；University of Texas at Dallas（德克萨斯大学达拉斯分校）
6D grasping in cluttered scenes is a longstanding problem in robotic manipulation. Open-loop manipulation pipelines may fail due to inaccurate state estimation, while most end-to-end grasping methods have not yet scaled to complex scenes with obstacles. In this work, we propose a new method for end-to-end learning of 6D grasping in cluttered scenes. Our hierarchical framework learns collision-free target-driven grasping based on partial point cloud observations. We learn an embedding space to encode expert grasping plans during training and a variational autoencoder to sample diverse grasping …
The process of planning views to observe a scene is known as the Next Best View (NBV) problem. Approaches often aim to obtain high-quality scene observations while reducing the number of views, travel distance and computational cost. Considering occlusions and scene coverage can significantly reduce the number of views and travel distance required to obtain an observation. Structured representations (e.g., a voxel grid or surface mesh) typically use raycasting to evaluate the visibility of represented structures but this is often computationally expensive. Unstructured representations (e.g., p…
HGS-Planner: Hierarchical Planning Framework for Active Scene Reconstruction Using 3D Gaussian Splatting
Academy for Engineering & Technology, Fudan University（复旦大学工程与技术学院）；Institute of Cyber-Systems and Control, College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院 cyber系统与控制研究所）；Department of Computer Science and Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）
Robot planning in partially observable domains is difficult, because a robot needs to estimate the current state and plan actions at the same time. When the domain includes many objects, reasoning about the objects and their relationships makes robot planning even more difficult. In this paper, we develop an algorithm called scene analysis for robot planning (SARP) that enables robots to reason with visual contextual information toward achieving long-term goals under uncertainty. SARP constructs scene graphs, a factored representation of objects and their relations, using images captured from …
