超越非专家演示：面向离线强化学习的结果驱动动作约束
Beyond Non-Expert Demonstrations: Outcome-Driven Action Constraint for Offline Reinforcement Learning
AI总结：
针对离线强化学习中的次优非专家数据问题，提出ODAF方法，通过结果驱动的保守奖励机制减少对行为策略动作分布的依赖，在MuJoCo等基准上有效容忍未见转移并提升轨迹拼接与学习能力。
摘要：
我们探讨了使用真实数据（特别是通过次优行为策略收集的非专家数据）进行离线强化学习的挑战。在此情况下，学习到的策略必须足够安全以应对分布偏移，同时保持足够的灵活性来处理离线数据中的非专家（糟糕）演示。为解决这一问题，我们引入了一种名为Outcome-Driven Action Flexibility (ODAF)的新方法，该方法旨在减少对行为策略经验动作分布的依赖，从而降低这些糟糕演示的负面影响。具体而言，我们开发了一种新的保守奖励机制，通过评估动作的结果是否满足安全要求（即保持在状态支持区域内）来应对分布偏移，而非仅依赖基于离线数据的动作似然度。除了理论论证外，我们还在广泛使用的MuJoCo和各种迷宫基准上提供了实证证据，表明使用不确定性量化技术实现的ODAF方法能有效容忍未见过的状态转移以改进“轨迹拼接”，同时增强智能体从真实非专家数据中学习的能力。
英文摘要：
We address the challenge of offline reinforcement learning using realistic data, specifically non-expert data collected through sub-optimal behavior policies. Under such circumstance, the learned policy must be safe enough to manage distribution shift while maintaining sufficient flexibility to deal with non-expert (bad) demonstrations from offline data.To tackle this issue, we introduce a novel method called Outcome-Driven Action Flexibility (ODAF), which seeks to reduce reliance on the empirical action distribution of the behavior policy, hence reducing the negative impact of those bad demonstrations.To be specific, a new conservative reward mechanism is developed to deal with distribution shift by evaluating actions according to whether their outcomes meet safety requirements - remaining within the state support area, rather than solely depending on the actions' likelihood based on offline data.Besides theoretical justification, we provide empirical evidence on widely used MuJoCo and various maze benchmarks, demonstrating that our ODAF method, implemented using uncertainty quantification techniques, effectively tolerates unseen transitions for improved "trajectory stitching," while enhancing the agent's ability to learn from realistic non-expert data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
