PlanTRansformer: Unified Prediction and Planning with Goal-conditioned Transformer
Department of Electrical Engineering and Information Technology, Intelligent Vehicles Lab (IVL), Munich University of Applied Science（电气工程与信息科技系，智能车辆实验室（IVL），慕尼黑应用科学大学）
Trajectory prediction and planning are fundamental yet disconnected components in autonomous driving. Prediction models forecast surrounding agent motion under unknown intentions, producing multimodal distributions, while planning assumes known ego objectives and generates deterministic trajectories. This mismatch creates a critical bottleneck: prediction lacks supervision for agent intentions, while planning requires this information. Existing prediction models, despite strong benchmarking performance, often remain disconnected from planning constraints such as collision avoidance and dynamic feasibility. We introduce Plan TRansformer (PTR), a unified Gaussian Mixture Transformer framework integrating goal-conditioned prediction, dynamic feasibility, interaction awareness, and lane-level topology reasoning. A teacher-student training strategy progressively masks surrounding agent commands during training to align with inference conditions where agent intentions are unavailable. PTR achieves 4.3%/3.5% improvement in marginal/joint mAP compared to the baseline Motion Transformer (MTR) and 15.5% planning error reduction at 5s horizon compared to GameFormer. The architecture-agnostic design enables application to diverse Transformer-based prediction models. Project Website: https://github.com/SelzerConst/PlanTRansformer
Bridging Predictive Uncertainty and Safe Action: Sample-Conditioned Differentiable Planning for Autonomous Driving
Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology（香港科学与技术大学机器人与自主系统方向）；Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（香港科学与技术大学电子与计算机工程系）；Department of Civil and Environmental Engineering, University of California, Los Angeles（加州大学洛杉矶分校土木与环境工程系）；School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与航空航天工程学院）
doPlan: A Variable-Horizon Dataset for Multi-Stage Language-Conditioned Planning in Autonomous Driving
Laboratory for Intelligent & Safe Automobiles (LISA), University of California, San Diego（加州大学圣迭戈分校智能与安全汽车实验室（LISA））；Machine Intelligence, Interaction, and Imagination (Mi) Laboratory, University of California, Merced（加州大学默塞德分校机器智能、交互与想象实验室（Mi））
DTPP: Differentiable Joint Conditional Prediction and Cost Evaluation for Tree Policy Planning in Autonomous Driving
School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与航天工程学院）；NVIDIA Research, NVIDIA Corporation（英伟达公司英伟达研究院）；Department of Aeronautics and Astronautics, Stanford University（斯坦福大学航空航天系）
MomWorld: Momentum-Aware Latent World Model for Long-Horizon Autonomous Driving
Nanyang Technological University（南洋理工大学）；Beijing Jiaotong University（北京交通大学）；North University of China（中北大学）；Dalian University of Technology（大连理工大学）；Tsinghua University（清华大学）
Predicting the motion of multiple agents is necessary for planning in dynamic environments. This task is challenging for autonomous driving since agents (e.g. vehicles and pedestrians) and their associated behaviors may be diverse and influence one another. Most prior work have focused on predicting independent futures for each agent based on all past motion, and planning against these independent predictions. However, planning against independent predictions can make it challenging to represent the future interaction possibilities between different agents, leading to sub-optimal planning. In …
PLAN-S: Bridging Planning with Latent Style Dynamics for Autonomous Driving World Models
Intelligent Transportation Thrust, Systems Hub, and Center of Seamless Connectivity & Connected Intelligence, The Hong Kong University of Science and Technology (Guangzhou)（智能交通 thrust、系统中心及无缝连接与智能连接研究院，香港科学与技术大学（广州））
Offline reinforcement learning (RL) provides a framework for learning decision-making from offline data and therefore constitutes a promising approach for real-world applications as automated driving. Self-driving vehicles (SDV) learn a policy, which potentially even outperforms the behavior in the sub-optimal data set. Especially in safety-critical applications as automated driving, explainability and transferability are key to success. This motivates the use of model-based offline RL approaches, which leverage planning. However, current state-of-the-art methods often neglect the influence of…
DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation
University of Technology Nuremberg (UTN)（纽伦堡工业大学）；Technical University of Munich (TUM)（慕尼黑工业大学）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；NVIDIA（英伟达）；Robotics Institute Germany（德国机器人研究所）
The ability to predict and plan into the future is fundamental for agents acting in the world. To reach a faraway goal, we predict trajectories at multiple timescales, first devising a coarse plan towards the goal and then gradually filling in details. In contrast, current learning approaches for visual prediction and planning fail on long-horizon tasks as they generate predictions (1) without considering goal information, and (2) at the finest temporal resolution, one step at a time. In this work we propose a framework for visual prediction and planning that is able to overcome both of these …
Planning methods can solve temporally extended sequential decision making problems by composing simple behaviors. However, planning requires suitable abstractions for the states and transitions, which typically need to be designed by hand. In contrast, model-free reinforcement learning (RL) can acquire behaviors from low-level inputs directly, but often struggles with temporally extended tasks. Can we utilize reinforcement learning to automatically form the abstractions needed for planning, thus obtaining the best of both approaches? We show that goal-conditioned policies learned with RL can b…
Structured-Diffuser: Diffusion with Task-Conditioned Structured Priors for Motion Planning
Institute for Robotics and Intelligent Machines (IRIM)（机器人与智能机器研究所）；Georgia Institute of Technology（佐治亚理工学院）；School of Computational Science and Engineering（计算科学与工程学院）
XFlowMP: Task-Conditioned Motion Fields for Generative Robot Planning with Schrodinger Bridges
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（莫扎德·本·扎耶德人工智能大学）；Automation & Control Institute (ACIN), TU Wien（自动化与控制研究所）；Austrian Institute of Technology (AIT) GmbH（奥地利技术研究所）
Offline Reinforcement learning (RL) has shown potent in many safe-critical tasks in robotics where exploration is risky and expensive. However, it still struggles to acquire skills in temporally extended tasks. In this paper, we study the problem of offline RL for temporally extended tasks. We propose a hierarchical planning framework, consisting of a low-level goal-conditioned RL policy and a high-level goal planner. The low-level policy is trained via offline RL. We improve the offline training to deal with out-of-distribution goals by a perturbed goal sampling process. The high-level planne…
NavThinker: Action-Conditioned World Models for Coupled Prediction and Planning in Social Navigation
The Hong Kong University of Science and Technology（香港科学与技术大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；National University of Singapore（新加坡国立大学）；University of Chinese Academy of Sciences（中国科学院大学）
Applications of Reinforcement Learning (RL) in robotics are often limited by high data demand. On the other hand, approximate models are readily available in many robotics scenarios, making model-based approaches like planning a data-efficient alternative. Still, the performance of these methods suffers if the model is imprecise or wrong. In this sense, the respective strengths and weaknesses of RL and model-based planners are. In the present work, we investigate how both approaches can be integrated into one framework that combines their strengths. We introduce Learning to Execute (L2E), whic…
Massachusetts Institute of Technology（麻省理工学院）；Toyota Research Institute（丰田研究院）
Risk-bounded motion planning is an important yet difficult problem for safety-critical tasks. While existing mathematical programming methods offer theoretical guarantees in the context of constrained Markov decision processes, they either lack scalability in solving larger problems or produce conservative plans. Recent advances in deep reinforcement learning improve scalability by learning policy networks as function approximators. In this paper, we propose an extension of soft actor critic model to estimate the execution risk of a plan through a risk critic and produce risk-bounded policies …
Long-horizon planning in realistic environments requires the ability to reason over sequential tasks in high-dimensional state spaces with complex dynamics. Classical motion planning algorithms, such as rapidly-exploring random trees, are capable of efficiently exploring large state spaces and computing long-horizon, sequential plans. However, these algorithms are generally challenged with complex, stochastic, and high-dimensional state spaces as well as in the presence of narrow passages, which naturally emerge in tasks that interact with the environment. Machine learning offers a promising s…
