谷歌地图中具备大规模可扩展性的逆强化学习
Massively Scalable Inverse Reinforcement Learning in Google Maps
AI总结：
针对逆强化学习难以支撑行星级路线推荐的问题，提出结合图压缩、空间并行等技术的RHIP算法，实现全球路线质量16-24%的提升，是目前最大规模的真实场景IRL研究。
摘要：
逆强化学习（Inverse Reinforcement Learning, IRL）为学习路线推荐场景下人类的潜在偏好提供了一个强大且通用的框架，但目前尚无方法能够成功解决包含数亿个状态和演示轨迹的行星级规模问题。本文中，我们提出了基于图压缩、空间并行化以及受特征向量算法关联启发的改进初始化条件的扩展技术。我们在路线规划场景下重新审视了经典IRL方法，并关键观察到：使用低成本的确定性规划器与高成本但鲁棒的随机策略之间存在权衡。我们将这一洞见应用于递推视界逆规划（Receding Horizon Inverse Planning, RHIP），这是经典IRL算法的一种新型泛化形式，可通过其规划视界对性能权衡进行细粒度控制。我们的贡献最终形成了一项在全球范围内将路线质量提升16-24%的策略，且据我们所知，这是迄今为止已发表的在真实场景中开展的最大规模IRL算法研究。最后，我们通过对关键组件进行消融研究、展示替代特征值求解器的负面结果，并指出通过IRL专属批处理策略进一步提升可扩展性的方向作为总结。
英文摘要：
Inverse reinforcement learning (IRL) offers a powerful and general framework for learning humans' latent preferences in route recommendation, yet no approach has successfully addressed planetary-scale problems with hundreds of millions of states and demonstration trajectories. In this paper, we introduce scaling techniques based on graph compression, spatial parallelization, and improved initialization conditions inspired by a connection to eigenvector algorithms. We revisit classic IRL methods in the routing context, and make the key observation that there exists a trade-off between the use of cheap, deterministic planners and expensive yet robust stochastic policies. This insight is leveraged in Receding Horizon Inverse Planning (RHIP), a new generalization of classic IRL algorithms that provides fine-grained control over performance trade-offs via its planning horizon. Our contributions culminate in a policy that achieves a 16-24% improvement in route quality at a global scale, and to the best of our knowledge, represents the largest published study of IRL algorithms in a real-world setting to date. We conclude by conducting an ablation study of key components, presenting negative results from alternative eigenvalue solvers, and identifying opportunities to further improve scalability via IRL-specific batching strategies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
