超立方体策略正则化框架用于离线强化学习
Hypercube Policy Regularization Framework for Offline Reinforcement Learning
AI总结：
本文提出超立方体策略正则化框架，通过允许智能体探索相似状态对应的动作，提升离线强化学习在低质量数据集中的性能。
摘要：
离线强化学习由于通过静态数据集学习策略而避免了智能体与环境的交互，因此受到学者的广泛关注。然而，通用强化学习方法在离线强化学习中无法获得满意的结果，因为数据集在训练过程中无法覆盖分布外的状态动作。为了解决这个问题，由于其简单和有效性，试图直接克隆静态数据集中使用的策略的策略正则化方法得到了大量研究。然而，策略约束方法使智能体选择静态数据集中的相应动作。这种约束通常过于保守，导致策略次优，尤其是在低质量的静态数据集中。在本文中，提出了一种超立方体策略正则化框架，该方法通过允许智能体探索静态数据集中相似状态对应的动作，缓解了策略约束方法的限制，从而在低质量数据集中提高了算法的有效性。此外，还从理论上证明了超立方体策略正则化框架可以有效提高原始算法的性能。此外，超立方体策略正则化框架与TD3-BC和Diffusion-QL结合，对D4RL数据集进行了实验，称为TD3-BC-C和Diffusion-QL-C。实验结果表明，在大多数D4RL环境中，TD3-BC-C和Diffusion-QL-C在近似时间内比IQL、CQL、TD3-BC和Diffusion-QL等最先进的算法表现更好。
英文摘要：
Offline reinforcement learning has received extensive attention from scholars because it avoids the interaction between the agent and the environment by learning a policy through a static dataset. However, general reinforcement learning methods cannot get satisfactory results in offline reinforcement learning due to the out-of-distribution state actions that the dataset cannot cover during training. To solve this problem, the policy regularization method that tries to directly clone policies used in static datasets has received numerous studies due to its simplicity and effectiveness. However, policy constraint methods make the agent choose the corresponding actions in the static dataset. This type of constraint is usually over-conservative, which results in suboptimal policies, especially in low-quality static datasets. In this paper, a hypercube policy regularization framework is proposed, this method alleviates the constraints of policy constraint methods by allowing the agent to explore the actions corresponding to similar states in the static dataset, which increases the effectiveness of algorithms in low-quality datasets. It was also theoretically demonstrated that the hypercube policy regularization framework can effectively improve the performance of original algorithms. In addition, the hypercube policy regularization framework is combined with TD3-BC and Diffusion-QL for experiments on D4RL datasets which are called TD3-BC-C and Diffusion-QL-C. The experimental results of the score demonstrate that TD3-BC-C and Diffusion-QL-C perform better than state-of-the-art algorithms like IQL, CQL, TD3-BC and Diffusion-QL in most D4RL environments in approximate time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
