arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.09838cs.LGcs.NE
收藏

缓解自适应列车调度中的稳定性-可塑性困境:基于课程驱动的持续DQN扩展

Mitigating the Stability-Plasticity Dilemma in Adaptive Train Scheduling with Curriculum-Driven Continual DQN Expansion

Goethe University(歌德大学);HessianAI(黑森人工智能公司)

Achref Jaziri, Etienne Künzel, Visvanathan Ramesh

AI总结:

针对自适应列车调度中的稳定性-可塑性困境,提出基于课程学习的持续DQN扩展算法(CDE),通过动态调整Q函数子空间并融合EWC与自适应激活函数,显著提升学习效率与适应性。

摘要:

持续学习智能体通过适应非平稳和动态环境,在保留先前获得的知识的同时,基于以往经验发展出日益复杂的行为。然而,扩展这些系统面临重大挑战,尤其是在平衡保留先前策略与使新策略适应当前环境之间的关系方面。这种平衡被称为稳定性-可塑性困境,在诸如列车调度问题等复杂多智能体领域中尤为突出,因为在这些领域中,环境和智能体行为不断变化,且搜索空间巨大。在这项工作中,我们提出利用课程学习来解决列车调度问题中的这些挑战。我们设计了一个由相邻技能组成的课程,这些技能相互构建以提高泛化性能。引入具有不同任务的课程会带来非平稳性,我们通过提出一种新算法来解决这一问题:持续深度Q网络(DQN)扩展(CDE)。我们的方法动态生成和调整Q函数子空间,以应对环境变化和任务需求。CDE通过EWC缓解灾难性遗忘,同时利用自适应有理激活函数确保高可塑性。实验结果表明,与强化学习基线和其它适用于持续学习的改编方法相比,我们的方法在学习效率和适应性方面有显著提升,凸显了我们的方法在自适应列车调度场景中管理稳定性-可塑性困境的潜力。

英文摘要:

A continual learning agent builds on previous experiences to develop increasingly complex behaviors by adapting to non-stationary and dynamic environments while preserving previously acquired knowledge. However, scaling these systems presents significant challenges, particularly in balancing the preservation of previous policies with the adaptation of new ones to current environments. This balance, known as the stability-plasticity dilemma, is especially pronounced in complex multi-agent domains such as the train scheduling problem, where environmental and agent behaviors are constantly changing, and the search space is vast. In this work, we propose addressing these challenges in the train scheduling problem using curriculum learning. We design a curriculum with adjacent skills that build on each other to improve generalization performance. Introducing a curriculum with distinct tasks introduces non-stationarity, which we address by proposing a new algorithm: Continual Deep Q-Network (DQN) Expansion (CDE). Our approach dynamically generates and adjusts Q-function subspaces to handle environmental changes and task requirements. CDE mitigates catastrophic forgetting through EWC while ensuring high plasticity using adaptive rational activation functions. Experimental results demonstrate significant improvements in learning efficiency and adaptability compared to RL baselines and other adapted methods for continual learning, highlighting the potential of our method in managing the stability-plasticity dilemma in the adaptive train scheduling setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑