引导式在线蒸馏：通过离线演示促进安全强化学习
Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration
AI总结：
提出GOLD框架，通过离线决策变换器策略引导在线安全强化学习蒸馏出轻量级策略，缓解保守性并提升安全关键场景决策性能。
摘要：
安全强化学习（Safe RL）旨在寻找一种能够在满足成本约束的同时获得高回报的策略。当从零开始学习时，安全强化学习智能体往往过于保守，这阻碍了探索并限制了整体性能。在许多现实任务中，例如自动驾驶，可获得大规模专家演示数据。我们认为，从离线数据中提取专家策略以引导在线探索是缓解保守问题的有前途的解决方案。大容量模型，例如决策变换器（DT），已被证明能够胜任离线策略学习。然而，在现实场景中收集的数据很少包含危险情况（例如碰撞），这使得策略难以学习安全概念。此外，这些庞大的策略网络无法满足在自动驾驶等现实任务中推理时的计算速度要求。为此，我们提出了引导式在线蒸馏（GOLD），一种离线到在线的安全强化学习框架。GOLD通过引导式在线安全强化学习训练，将离线DT策略蒸馏为轻量级策略网络，其性能优于离线DT策略和在线安全强化学习算法。在基准安全强化学习任务和基于Waymo开放运动数据集（WOMD）的现实驾驶任务中的实验表明，GOLD能够成功蒸馏轻量级策略，并解决具有挑战性的安全关键场景中的决策问题。
英文摘要：
Safe Reinforcement Learning (RL) aims to find a policy that achieves high rewards while satisfying cost constraints. When learning from scratch, safe RL agents tend to be overly conservative, which impedes exploration and restrains the overall performance. In many realistic tasks, e.g. autonomous driving, large-scale expert demonstration data are available. We argue that extracting expert policy from offline data to guide online exploration is a promising solution to mitigate the conserveness issue. Large-capacity models, e.g. decision transformers (DT), have been proven to be competent in offline policy learning. However, data collected in real-world scenarios rarely contain dangerous cases (e.g., collisions), which makes it prohibitive for the policies to learn safety concepts. Besides, these bulk policy networks cannot meet the computation speed requirements at inference time on real-world tasks such as autonomous driving. To this end, we propose Guided Online Distillation (GOLD), an offline-to-online safe RL framework. GOLD distills an offline DT policy into a lightweight policy network through guided online safe RL training, which outperforms both the offline DT policy and online safe RL algorithms. Experiments in both benchmark safe RL tasks and real-world driving tasks based on the Waymo Open Motion Dataset (WOMD) demonstrate that GOLD can successfully distill lightweight policies and solve decision-making problems in challenging safety-critical scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
