arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.10458cs.AIcs.LG
收藏

Found-RL: 基于基础模型的强化学习用于自动驾驶

Found-RL: foundation model-enhanced reinforcement learning for autonomous driving

Purdue University(普渡大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);University of Texas at Austin(德克萨斯大学奥斯汀分校)

Yansong Qu, Zihao Sheng, Zilin Huang, Jiancong Chen, Yuhao Luo, Tianyi Wang, Yiheng Feng, Samuel Labi, Sikai Chen

AI总结:

Found-RL通过异步批量推理和多样化监督机制,提升自动驾驶中强化学习的效率与实时性。

摘要:

强化学习(RL)已成为端到端自动驾驶(AD)的主导范式。然而,RL在复杂场景中存在样本效率低和语义可解释性差的问题。基础模型,特别是视觉-语言模型(VLMs),可以通过提供丰富、上下文感知的知识来缓解这一问题,但其高推理延迟阻碍了在高频RL训练循环中的部署。为弥合这一差距,我们提出了Found-RL,一个专门用于高效增强RL用于自动驾驶的平台。核心创新是异步批量推理框架,它将重型VLM推理与模拟循环解耦,有效解决了延迟瓶颈,以支持实时学习。我们引入了多样化的监督机制:价值边际正则化(VMR)和优势加权动作指导(AWAG)以有效地将专家样式的VLM动作建议蒸馏到RL策略中。此外,我们采用高吞吐量CLIP进行密集奖励塑造。我们通过条件对比动作对齐来解决CLIP的动态盲区,该方法将提示条件化于离散化速度/命令,并产生一个归一化、基于边际的奖励来自上下文特定的动作锚点评分。Found-RL提供了一个端到端的流程用于微调VLM的集成,并表明一个轻量级的RL模型可以实现接近VLM性能,与十亿参数VLMs相比,同时维持实时推理(约500 FPS)。代码、数据和模型将在https://github.com/ys-qu/found-rl公开。

英文摘要:

Reinforcement Learning (RL) has emerged as a dominant paradigm for end-to-end autonomous driving (AD). However, RL suffers from sample inefficiency and a lack of semantic interpretability in complex scenarios. Foundation Models, particularly Vision-Language Models (VLMs), can mitigate this by offering rich, context-aware knowledge, yet their high inference latency hinders deployment in high-frequency RL training loops. To bridge this gap, we present Found-RL, a platform tailored to efficiently enhance RL for AD using foundation models. A core innovation is the asynchronous batch inference framework, which decouples heavy VLM reasoning from the simulation loop, effectively resolving latency bottlenecks to support real-time learning. We introduce diverse supervision mechanisms: Value-Margin Regularization (VMR) and Advantage-Weighted Action Guidance (AWAG) to effectively distill expert-like VLM action suggestions into the RL policy. Additionally, we adopt high-throughput CLIP for dense reward shaping. We address CLIP's dynamic blindness via Conditional Contrastive Action Alignment, which conditions prompts on discretized speed/command and yields a normalized, margin-based bonus from context-specific action-anchor scoring. Found-RL provides an end-to-end pipeline for fine-tuned VLM integration and shows that a lightweight RL model can achieve near-VLM performance compared with billion-parameter VLMs while sustaining real-time inference (approx. 500 FPS). Code, data, and models will be publicly available at https://github.com/ys-qu/found-rl.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑