本文解决在部分可观测环境下训练强化学习策略的问题，通过在训练期间独有的特权规划器代理来实现。我们将此问题形式化为部分可观测马尔可夫决策过程（POMDP），其中具有近似动态模型和特权状态信息的规划器代理指导仅观测真实状态损失投影的学习代理。为实现此框架，我们引入了任何时间可行的模型预测控制（MPC）算法作为规划器代理。对于学习代理，我们提出了一种名为Planner-to-Policy Soft Actor-Critic（P2P-SAC）的方法，通过蒸馏规划器代理的特权知识来缓解部分可观测性，从而提高样本效率和最终策略性能。我们通过NVIDIA Isaac Lab仿真和Unitree Go2四足机器人在复杂障碍环境中的真实部署验证了该方法。
英文摘要：
This paper addresses the problem of training a reinforcement learning (RL) policy under partial observability by exploiting a privileged, anytime-feasible planner agent available exclusively during training. We formalize this as a Partially Observable Markov Decision Process (POMDP) in which a planner agent with access to an approximate dynamical model and privileged state information guides a learning agent that observes only a lossy projection of the true state. To realize this framework, we introduce an anytime-feasible Model Predictive Control (MPC) algorithm that serves as the planner agent. For the learning agent, we propose Planner-to-Policy Soft Actor-Critic (P2P-SAC), a method that distills the planner agent's privileged knowledge to mitigate partial observability and thereby improve both sample efficiency and final policy performance. We support this framework with rigorous theoretical analysis. Finally, we validate our approach in simulation using NVIDIA Isaac Lab and successfully deploy it on a real-world Unitree Go2 quadruped navigating complex, obstacle-rich environments.
UC Berkeley（加州大学伯克利分校）；Honda Research Institute（本田研究所）
A human-centered robot needs to reason about the cognitive limitation and potential irrationality of its human partner to achieve seamless interactions. This paper proposes an anytime game-theoretic planner that integrates iterative reasoning models, a partially observable Markov decision process, and chance-constrained Monte-Carlo belief tree search for robot behavioral planning. Our planner enables a robot to safely and actively reason about its human partner's latent cognitive states (bounded intelligence and irrationality) in real-time to maximize its utility better. We validate our approa…
Carnegie Mellon University（卡内基梅隆大学）；Massachusetts Institute of Technology（麻省理工学院）；University of Toronto（多伦多大学）；University of Waterloo（滑铁卢大学）；University of Toronto Institute for Aerospace Studies（多伦多大学航空航天研究所）
Path planning in robotics often requires finding high-quality solutions to continuously valued and/or high-dimensional problems. These problems are challenging and most planning algorithms instead solve simplified approximations. Popular approximations include graphs and random samples, as respectively used by informed graph-based searches and anytime sampling-based planners. Informed graph-based searches, such as A*, traditionally use heuristics to search a priori graphs in order of potential solution quality. This makes their search efficient but leaves their performance dependent on the cho…
Path planning is an active area of research essential for many applications in robotics. Popular techniques include graph-based searches and sampling-based planners. These approaches are powerful but have limitations. This paper continues work to combine their strengths and mitigate their limitations using a unified planning paradigm. It does this by viewing the path planning problem as the two subproblems of search and approximation and using advanced graph-search techniques on a sampling-based approximation. This perspective leads to Advanced BIT*. ABIT* combines truncated anytime graph-base…
Anytime motion planners are widely used in robotics. However, the relationship between their solution quality and computation time is not well understood, and thus, determining when to quit planning and start execution is unclear. In this paper, we address the problem of deciding when to stop deliberation under bounded computational capacity, so called meta-reasoning, for anytime motion planning. We propose data-driven learning methods, model-based and model-free meta-reasoning, that are applicable to different environment distributions and agnostic to the choice of anytime motion planners. As…
Estimated Informed Anytime Search for Sampling-Based Planning via Adaptive Sampler
Chair of Robotics, Artificial Intelligence and Real-time Systems, TUM School of Computation, Information and Technology (CIT), Technical University of Munich（机器人、人工智能与实时系统教授会，技术大学慕尼黑计算、信息与技术学院）；Chair of Robotics and Systems Intelligence, Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich（机器人与系统智能教授会，慕尼黑机器人与机器智能研究所，技术大学慕尼黑）；State Key Laboratory for Novel Software Technology and the School of Science and Technology, Nanjing University (Suzhou Campus)（新型软件技术国家重点实验室和南京大学（苏州校区）科学与技术学院）；School of Control Science and Engineering, Shandong University（控制科学与工程学院，山东大学）
We focus on the task of object manipulation to an arbitrary goal pose, in which a robot is supposed to pick an assigned object to place at the goal position with a specific orientation. However, limited by the execution space of the manipulator with gripper, one-step picking, moving and releasing might be failed, where a reorientation object pose is required as a transition. In this paper, we propose a learning-driven anytime prioritized search-based solver to find a feasible solution with low path cost in a short time. In our work, the problem is formulated as a hierarchical learning problem,…
Many methods have been developed for planning the motion of robotic arms for picking and placing, ranging from local optimization to global search techniques, which are effective for sparsely placed objects. Dense clutter, however, still adversely affects the success rate, computation times, and quality of solutions in many real-world setups. The current work integrates tools from existing methodologies and proposes a framework that achieves high success ratio in clutter with anytime performance. The idea is to first explore the lower dimensional end effector's task space efficiently by ignori…
TU Berlin（柏林工业大学）；BITS Pilani（比尔拉科学技术学院皮拉尼分校）；KTH Royal Institute of Technology（KTH皇家理工学院）
Optimal sampling based motion planning and trajectory optimization are two competing frameworks to generate optimal motion plans. Both frameworks have complementary properties: Sampling based planners are typically slow to converge, but provide optimality guarantees. Trajectory optimizers, however, are typically fast to converge, but do not provide global optimality guarantees in nonconvex problems, e.g. scenarios with obstacles. To achieve the best of both worlds, we introduce a new planner, BITKOMO, which integrates the asymptotically optimal Batch Informed Trees (BIT*) planner with the K-Or…
University of Cambridge（剑桥大学）；DeepMind（深度思维）；Mila（米拉人工智能研究所）；Université de Montréal（蒙特利尔大学）
Neural algorithmic reasoning studies the problem of learning algorithms with neural networks, especially with graph architectures. A recent proposal, XLVIN, reaps the benefits of using a graph neural network that simulates the value iteration algorithm in deep reinforcement learning agents. It allows model-free planning without access to privileged information about the environment, which is usually unavailable. However, XLVIN only supports discrete action spaces, and is hence nontrivially applicable to most tasks of real-world interest. We expand XLVIN to continuous action spaces by discretiz…
Flexible Informed Trees (FIT*): Adaptive Batch-Size Approach in Informed Sampling-Based Path Planning
Department of Informatics, Technical University of Munich（信息学院，慕尼黑技术大学）；Department of Technology & Innovation, KION Group Linde Material Handling GmbH（技术与创新学院，KION集团Linde物料处理 GmbH）
Greedy Heuristics for Sampling-Based Motion Planning in High-Dimensional State Spaces
University of Toronto Institute for Aerospace Studies（多伦多大学航空航天研究所）；Ton Duc Thang University（越南 ton duc thang 大学（东德盛大学））；Singapore University of Technology and Design（新加坡科技设计大学）
Integrated Task and Motion Planning (TMP) provides a promising class of approaches for solving robot planning problems with intricate symbolic and geometric constraints. However, the practical usefulness of TMP planners is limited by their need for symbolic abstractions of robot actions, which are difficult to construct even for experts. We propose an approach to automatically construct and continuously improve a symbolic abstraction of a robot action via observations of the robot performing the action. This approach, called automatic abstraction repair, allows symbolic abstractions to be init…
Autonomous Block Assembly for Boom Cranes with Passive Joint Dynamics: Integrated Vision MPC Control
Automation & Control Institute (ACIN), TU Wien（自动化与控制研究所（ACIN），维也纳技术大学）；Center for Vision, Automation & Control, AIT Austrian Institute of Technology GmbH（视觉、自动化与控制中心，奥地利技术研究所有限公司）
