具有实时预算约束的安全离线强化学习
Safe Offline Reinforcement Learning with Real-Time Budget Constraints
AI总结：
针对离线安全RL中策略需实时响应动态安全预算的问题，提出TREBI，利用轨迹分布建模与扩散模型规划求解，并给出误差界理论保证及仿真和大规模广告应用验证。
摘要：
为了推动强化学习（RL）在现实世界中的安全部署，安全RL研究近年来取得了显著进展。然而，文献中的大多数现有工作仍聚焦于在线设定，在训练过程中可能引发对安全预算的危险违反。此外，在许多现实应用中，所学策略需要实时响应动态确定的安全预算（即约束阈值）。本文针对离线设定下的上述实时预算约束问题，提出了基于轨迹的实时预算推断（TREBI）这一新颖解决方案，从轨迹分布的角度对该问题进行建模，并通过扩散模型规划加以求解。在理论方面，我们证明了离线设定下回合奖励与成本估计的误差界，从而为TREBI提供了性能保证。在广泛的仿真任务以及一个现实世界大规模广告应用上的实证结果，证明了TREBI在离线设定下解决实时预算约束问题的能力。
英文摘要：
Aiming at promoting the safe real-world deployment of Reinforcement Learning (RL), research on safe RL has made significant progress in recent years. However, most existing works in the literature still focus on the online setting where risky violations of the safety budget are likely to be incurred during training. Besides, in many real-world applications, the learned policy is required to respond to dynamically determined safety budgets (i.e., constraint threshold) in real time. In this paper, we target at the above real-time budget constraint problem under the offline setting, and propose Trajectory-based REal-time Budget Inference (TREBI) as a novel solution that models this problem from the perspective of trajectory distribution and solves it through diffusion model planning. Theoretically, we prove an error bound of the estimation on the episodic reward and cost under the offline setting and thus provide a performance guarantee for TREBI. Empirical results on a wide range of simulation tasks and a real-world large-scale advertising application demonstrate the capability of TREBI in solving real-time budget constraint problems under offline settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
