arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.07795cs.LGcs.AI
收藏

基于轻量Transformer的交通信号控制:一种离线到在线的强化学习方法

Traffic Signal Control Using Lightweight Transformers: An Offline-to-Online RL Approach

McGill University(麦吉尔大学)

Xingshuai Huang, Di Wu, Benoit Boulet

AI总结:

针对现有强化学习交通信号控制方法样本效率低、计算需求高的问题,提出基于轻量决策Transformer的DTLight方法,结合知识蒸馏与适配器模块实现离线预训练+低成本在线微调,性能优于现有最优在线RL方法,同时发布专用数据集DTRL。

摘要:

高效的交通信号控制对于缓解交通拥堵、提升整体交通效率至关重要。交通流的动态特性促使研究人员将强化学习(Reinforcement Learning, RL)应用于交通信号控制(traffic signal control, TSC)领域。与传统方法相比,基于RL的解决方案展现出了更优的性能,但这类方案存在样本效率低、计算需求高的问题,限制了其在现实世界中的落地应用。\n本文提出了DTLight——一种简洁且高效的基于轻量决策Transformer(Decision Transformer)的TSC方法,能够从易获取的离线数据集中学习策略。DTLight创新性地利用知识蒸馏技术,从训练好的大型教师模型中学习得到轻量控制器,以降低部署时的计算开销。此外,该方法集成了适配器模块,减少了微调所需的成本,使得DTLight在实际部署中仅需极少的计算量和少量微调步骤即可完成在线适配,具备很强的实用性。同时,本文还对DTLight做了进一步优化,使其更适用于现实场景中的TSC问题。\n在合成场景和真实场景下开展的大量实验表明,仅用离线数据集预训练的DTLight在大多数场景中性能优于当前最优的在线RL方法。实验结果还显示,经过在线微调后,DTLight的性能较最优的在线RL基线方法进一步提升了最高42.6%。本文还发布了专门面向离线RL的TSC数据集(命名为DTRL),相关数据集和代码已公开。

英文摘要:

Efficient traffic signal control is critical for reducing traffic congestion and improving overall transportation efficiency. The dynamic nature of traffic flow has prompted researchers to explore Reinforcement Learning (RL) for traffic signal control (TSC). Compared with traditional methods, RL-based solutions have shown preferable performance. However, the application of RL-based traffic signal controllers in the real world is limited by the low sample efficiency and high computational requirements of these solutions. In this work, we propose DTLight, a simple yet powerful lightweight Decision Transformer-based TSC method that can learn policy from easily accessible offline datasets. DTLight novelly leverages knowledge distillation to learn a lightweight controller from a well-trained larger teacher model to reduce implementation computation. Additionally, it integrates adapter modules to mitigate the expenses associated with fine-tuning, which makes DTLight practical for online adaptation with minimal computation and only a few fine-tuning steps during real deployment. Moreover, DTLight is further enhanced to be more applicable to real-world TSC problems. Extensive experiments on synthetic and real-world scenarios show that DTLight pre-trained purely on offline datasets can outperform state-of-the-art online RL-based methods in most scenarios. Experiment results also show that online fine-tuning further improves the performance of DTLight by up to 42.6% over the best online RL baseline methods. In this work, we also introduce Datasets specifically designed for TSC with offline RL (referred to as DTRL). Our datasets and code are publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑