高效的交通信号控制对于缓解交通拥堵、提升整体交通效率至关重要。交通流的动态特性促使研究人员将强化学习(Reinforcement Learning, RL)应用于交通信号控制(traffic signal control, TSC)领域。与传统方法相比,基于RL的解决方案展现出了更优的性能,但这类方案存在样本效率低、计算需求高的问题,限制了其在现实世界中的落地应用。\n本文提出了DTLight——一种简洁且高效的基于轻量决策Transformer(Decision Transformer)的TSC方法,能够从易获取的离线数据集中学习策略。DTLight创新性地利用知识蒸馏技术,从训练好的大型教师模型中学习得到轻量控制器,以降低部署时的计算开销。此外,该方法集成了适配器模块,减少了微调所需的成本,使得DTLight在实际部署中仅需极少的计算量和少量微调步骤即可完成在线适配,具备很强的实用性。同时,本文还对DTLight做了进一步优化,使其更适用于现实场景中的TSC问题。\n在合成场景和真实场景下开展的大量实验表明,仅用离线数据集预训练的DTLight在大多数场景中性能优于当前最优的在线RL方法。实验结果还显示,经过在线微调后,DTLight的性能较最优的在线RL基线方法进一步提升了最高42.6%。本文还发布了专门面向离线RL的TSC数据集(命名为DTRL),相关数据集和代码已公开。
英文摘要:
Efficient traffic signal control is critical for reducing traffic congestion and improving overall transportation efficiency. The dynamic nature of traffic flow has prompted researchers to explore Reinforcement Learning (RL) for traffic signal control (TSC). Compared with traditional methods, RL-based solutions have shown preferable performance. However, the application of RL-based traffic signal controllers in the real world is limited by the low sample efficiency and high computational requirements of these solutions. In this work, we propose DTLight, a simple yet powerful lightweight Decision Transformer-based TSC method that can learn policy from easily accessible offline datasets. DTLight novelly leverages knowledge distillation to learn a lightweight controller from a well-trained larger teacher model to reduce implementation computation. Additionally, it integrates adapter modules to mitigate the expenses associated with fine-tuning, which makes DTLight practical for online adaptation with minimal computation and only a few fine-tuning steps during real deployment. Moreover, DTLight is further enhanced to be more applicable to real-world TSC problems. Extensive experiments on synthetic and real-world scenarios show that DTLight pre-trained purely on offline datasets can outperform state-of-the-art online RL-based methods in most scenarios. Experiment results also show that online fine-tuning further improves the performance of DTLight by up to 42.6% over the best online RL baseline methods. In this work, we also introduce Datasets specifically designed for TSC with offline RL (referred to as DTRL). Our datasets and code are publicly available.
A Fully Data-Driven Approach for Realistic Traffic Signal Control Using Offline Reinforcement Learning
Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));School of Vehicle and Mobility, Tsinghua University(清华大学车辆与出行学院);Baidu Inc.(百度公司);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Minimal Ingredients for Reward Assignment from Expert Demonstrations
New York University(纽约大学);New York University Abu Dhabi(纽约大学阿布扎赫尔分校);Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning, NYU Shanghai(上海前沿科学中心(人工智能与深度学习))
Test-Time Learning and Inference-Time Deliberation for Efficiency-First Offline Reinforcement Learning in Care Coordination and Population Health Management
DLM: Unified Decision Language Models for Offline Multi-Agent Sequential Decision Making
Department of Control Science and Engineering, Tongji University, Shanghai 201804, China(同济大学控制科学与工程学院);Shanghai Institute of Intelligent Science and Technology(上海智能科学研究院);Shanghai Research Institute for Intelligent Autonomous Systems(上海智能自主系统研究院);State Key Laboratory of Autonomous Intelligent Unmanned Systems, Shanghai 200120, China(自主智能无人系统国家重点实验室)
DiDrive: A Risk-Aware Hierarchical Diffusion Framework for Safe Offline Reinforcement Learning in Autonomous Driving
College of Computer and Data Science, Fuzhou University(福州大学计算机与数据科学学院);Maynooth International College of Engineering, Fuzhou University(福州大学梅努斯国际工程学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
A Lightweight and Accurate Spatial-Temporal Transformer for Traffic Forecasting
The Hong Kong University of Science and Technology(香港科技大学);Chongqing University(重庆大学);National Chung Hsing University(国立中兴大学);National Yang Ming Chiao Tung University(国立阳明交通大学)
A Multi-View Coupled Tensor Decomposition for Lightweight Online Adaptive Traffic Prediction
School of Mathematics and Statistics, Central China Normal University(华中师范大学数学与统计学学院);School of Mathematics and Statistics, Beijing Institute of Technology(北京理工大学数学与统计学学院);State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院数学科学国家重点实验室);School of Mathematical Sciences, University of Chinese Academy of Sciences(中国科学院大学数学科学学院)
SCOPE: A Lightweight-training LLM Framework for Air Traffic Control Readback Monitoring
Department of Mechanical and Aerospace Engineering, The Hong Kong University of Science and Technology(香港科学与技术大学机械与航空航天工程系);School of Electronic and Information Engineering, Beihang University(北航电子与信息工程学院);State Key Laboratory of CNS/ATM(国家空管自动化系统实验室)