MAGNIFIED: 多模态大语言模型的强化学习微调用于运动规划
MAGNIFIED: RL Fine-tuning of Multimodal Large Language Models for Motion Planning
AI总结：
提出MAGNIFIED方法，通过强化学习微调多模态大语言模型，利用令牌级奖励优化规划目标，在Waymo数据集上显著降低重叠率和偏离道路率。
摘要：
多模态大语言模型（MLLMs）在语义理解和常识推理方面展现出卓越能力，使其成为解决自动驾驶规划问题的有前景候选。然而，MLLMs预训练和监督微调中传统使用的下一个令牌文本预测目标可能无法满足自动驾驶车辆的规划目标。下一个令牌预测目标仅仅鼓励文本中的逐令牌模仿，通常不考虑多步后果以及与关键规划考虑（如为其他道路参与者让出空间）的一致性。为了克服这些限制，我们提出了一种强化学习微调方法MAGNIFIED，通过从令牌级奖励中学习，使基于MLLM的驾驶代理与规划目标对齐。通过将预测的令牌序列映射到相应的车辆轨迹并从规划奖励中学习，MAGNIFIED优化了真正的规划目标，而不仅仅关注令牌预测准确性，使模型能够超越简单模仿，细化对规划任务的理解。我们在Waymo开放运动数据集上验证了我们的方法，采用了一种新颖的设置，将栅格化鸟瞰图和令牌化轨迹作为输入，并以规划为导向输出。初始的SFT阶段建立了强大的基线，能够以文本形式输出X-Y坐标序列的规划轨迹，而随后的强化学习微调相对于SFT基线显著提升了规划性能（重叠率降低超过10.5%，偏离道路率降低38.9%），突显了RLFT在MLLMs上实现与合规、舒适和高效驾驶更一致的车辆规划的潜力。
英文摘要：
Multi-modal Large Language Models (MLLMs) have demonstrated remarkable capabilities in semantic understanding and common sense reasoning, making them promising candidates for solving planning problems in autonomous driving. However, the next-token text prediction objectives traditionally used in pre-training and supervised fine-tuning (SFT) of MLLMs may fall short of fulfilling the planning objectives for autonomous vehicles. The next-token prediction objective merely encourages per-token imitation in text, often irrespective of multi-step consequences and the alignment with crucial planning considerations such as giving space to other road actors. To overcome these limitations, we propose a reinforcement learning fine-tuning (RLFT) approach, MAGNIFIED, that aligns the MLLM-based driving agent with planning objectives by learning from token-level rewards. By mapping a sequence of predicted tokens to corresponding vehicle trajectories and learning from planning rewards, MAGNIFIED optimizes for the true planning objectives rather than focusing solely on token prediction accuracy, enabling the model to refine its understanding of the planning task beyond simple imitation. We validate our approach on the Waymo Open Motion Dataset with a novel setup incorporating rasterized birds-eye views and tokenized trajectories as inputs and planning-oriented outputs. An initial SFT phase establishes a strong baseline in outputting plan trajectories as sequences of X-Y coordinates in text, while subsequent RL fine-tuning substantially enhances planning performance relative to the SFT baseline (demonstrating over a 10.5% reduction in overlap rate and a 38.9% reduction in off-road rate), underscoring the potential of RLFT on MLLMs to achieve vehicle planning that is better aligned with compliant, comfortable, and efficient driving.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
