arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2505.03344cs.ROcs.LG
收藏

RIFT:面向真实且可控交通仿真的群体相对强化学习微调

RIFT: Group-Relative RL Fine-Tuning for Realistic and Controllable Traffic Simulation

School of Vehicle and Mobility, Tsinghua University(车辆与移动学院,清华大学)

Keyu Chen, Wenchao Sun, Hao Cheng, Sifa Zheng

AI总结:

本文提出双阶段自动驾驶交通仿真框架,先进行模仿学习预训练,再通过群体相对强化学习微调策略RIFT,在保留轨迹真实性与路线可控性的同时增强风格可控性并缓解协变量偏移。

摘要:

在自动驾驶中,闭环交通仿真同时实现真实性与可控性仍是一项关键挑战。基于数据集的方法能够复现真实轨迹,但在闭环部署中会遭受协变量偏移问题,加之简化的动力学模型进一步降低了可靠性。相反,基于物理的仿真方法增强了可靠且可控的闭环交互,但往往缺乏专家演示,从而损害了真实性。为应对这些挑战,我们引入了一个以自动驾驶车辆为中心的双阶段仿真框架:首先在数据驱动仿真器中进行模仿学习预训练,以捕获轨迹级真实性和路线级可控性;随后在基于物理的仿真器中进行强化学习微调,以增强风格级可控性并缓解协变量偏移。在微调阶段,我们提出了RIFT,一种新颖的群体相对强化学习微调策略,该策略通过群体相对公式化评估所有候选模态,并采用代理目标进行稳定优化,在保留模仿学习预训练所继承的轨迹级真实性和路线级可控性的同时,增强风格级可控性并缓解协变量偏移。大量实验表明,RIFT提升了交通仿真的真实性与可控性,同时在闭环评估中暴露了现代自动驾驶系统的局限性。项目页面:https://currychen77.github.io/RIFT/

英文摘要:

Achieving both realism and controllability in closed-loop traffic simulation remains a key challenge in autonomous driving. Dataset-based methods reproduce realistic trajectories but suffer from covariate shift in closed-loop deployment, compounded by simplified dynamics models that further reduce reliability. Conversely, physics-based simulation methods enhance reliable and controllable closed-loop interactions but often lack expert demonstrations, compromising realism. To address these challenges, we introduce a dual-stage AV-centric simulation framework that conducts imitation learning pre-training in a data-driven simulator to capture trajectory-level realism and route-level controllability, followed by reinforcement learning fine-tuning in a physics-based simulator to enhance style-level controllability and mitigate covariate shift. In the fine-tuning stage, we propose RIFT, a novel group-relative RL fine-tuning strategy that evaluates all candidate modalities through group-relative formulation and employs a surrogate objective for stable optimization, enhancing style-level controllability and mitigating covariate shift while preserving the trajectory-level realism and route-level controllability inherited from IL pre-training. Extensive experiments demonstrate that RIFT improves realism and controllability in traffic simulation while simultaneously exposing the limitations of modern AV systems in closed-loop evaluation. Project Page: https://currychen77.github.io/RIFT/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑