(Re)$^2$H2O：基于反向正则化混合离线-在线强化学习的自动驾驶场景生成
(Re)$^2$H2O: Autonomous Driving Scenario Generation via Reversely Regularized Hybrid Offline-and-Online Reinforcement Learning
AI总结：
针对自动驾驶测试场景生成需求，提出(Re)$^2$H2O反向正则化混合离线-在线强化学习方法，结合真实与仿真数据优势，生成更多样且具对抗性的高风险场景，可泛化多模型并用于AV性能微调。
摘要：
自动驾驶及其广泛应用长期以来前景广阔。然而，若缺乏可靠且全面的测试流程，行业不仅难以量产自动驾驶汽车（AV），普通公众和政策制定者也无法信服并接受这类创新。生成对自动驾驶汽车构成重大挑战的安全关键场景，是测试工作中至关重要的第一步。真实世界数据集包含自然驾驶行为，但这类行为过于安全；而仿真则可不受限制地探索多样且激进的交通场景。但反过来，仿真中更高维度的搜索空间，若没有真实世界数据分布作为隐式约束，就无法实现高效的场景生成。为结合二者优势，同时从离线真实世界数据和在线仿真数据中学习生成场景似乎颇具吸引力。因此，我们定制了反向正则化混合离线-在线（(Re)$^2$H2O）强化学习方案，额外对真实世界数据的Q值施加惩罚，并对仿真数据的Q值给予奖励，以此确保生成的场景既具备多样性又具有对抗性。通过大量实验，我们的方案被证明能比同类基准方法生成更多高风险场景，且可泛化适配多种自动驾驶模型。此外，这些生成的场景也被证实能够用于微调自动驾驶汽车的性能。
英文摘要：
Autonomous driving and its widespread adoption have long held tremendous promise. Nevertheless, without a trustworthy and thorough testing procedure, not only does the industry struggle to mass-produce autonomous vehicles (AV), but neither the general public nor policymakers are convinced to accept the innovations. Generating safety-critical scenarios that present significant challenges to AV is an essential first step in testing. Real-world datasets include naturalistic but overly safe driving behaviors, whereas simulation would allow for unrestricted exploration of diverse and aggressive traffic scenarios. Conversely, higher-dimensional searching space in simulation disables efficient scenario generation without real-world data distribution as implicit constraints. In order to marry the benefits of both, it seems appealing to learn to generate scenarios from both offline real-world and online simulation data simultaneously. Therefore, we tailor a Reversely Regularized Hybrid Offline-and-Online ((Re)$^2$H2O) Reinforcement Learning recipe to additionally penalize Q-values on real-world data and reward Q-values on simulated data, which ensures the generated scenarios are both varied and adversarial. Through extensive experiments, our solution proves to produce more risky scenarios than competitive baselines and it can generalize to work with various autonomous driving models. In addition, these generated scenarios are also corroborated to be capable of fine-tuning AV performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
