基于epigraph的流匹配：安全且高效的离线强化学习
Epigraph-Guided Flow Matching for Safe and Performant Offline Reinforcement Learning
AI总结：
EpiFlow通过epigraph引导的流匹配方法，在安全关键任务中实现了高效且安全的离线强化学习。
摘要：
离线强化学习（RL）提供了一种有吸引力的范式，用于在不涉及在线探索风险的情况下训练自主系统，特别是在安全关键领域。然而，从固定数据集中同时实现强大的安全性和性能仍然具有挑战性。现有的安全离线RL方法通常依赖于软约束，允许违反、引入过度保守性或难以平衡安全、奖励优化和遵循数据分布。为此，我们提出了Epigraph-Guided Flow Matching（EpiFlow），一个将安全离线RL制定为状态约束最优控制问题的框架，以共同优化安全性和性能。我们学习了一个从最优控制问题的epigraph重述中衍生出的可行性价值函数，从而避免了先前工作中常见的解耦目标或事后过滤。通过基于此epigraph价值函数重新加权行为分布并拟合生成策略，可以实现高效且分布一致的采样。在各种安全关键任务中，包括Safety-Gymnasium基准测试，EpiFlow实现了与近零经验安全违规相媲美的竞争力回报，证明了epigraph引导策略合成的有效性。
英文摘要：
Offline reinforcement learning (RL) provides a compelling paradigm for training autonomous systems without the risks of online exploration, particularly in safety-critical domains. However, jointly achieving strong safety and performance from fixed datasets remains challenging. Existing safe offline RL methods often rely on soft constraints that allow violations, introduce excessive conservatism, or struggle to balance safety, reward optimization, and adherence to the data distribution. To address this, we propose Epigraph-Guided Flow Matching (EpiFlow), a framework that formulates safe offline RL as a state-constrained optimal control problem to co-optimize safety and performance. We learn a feasibility value function derived from an epigraph reformulation of the optimal control problem, thereby avoiding the decoupled objectives or post-hoc filtering common in prior work. Policies are synthesized by reweighting the behavior distribution based on this epigraph value function and fitting a generative policy via flow matching, enabling efficient, distribution-consistent sampling. Across various safety-critical tasks, including Safety-Gymnasium benchmarks, EpiFlow achieves competitive returns with near-zero empirical safety violations, demonstrating the effectiveness of epigraph-guided policy synthesis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
