近端动作替换用于离线强化学习中的行为克隆Actor-Critic
Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learning
AI总结：
本文提出近端动作替换方法，解决离线强化学习中行为克隆导致的性能瓶颈问题，通过替换亚优动作提升Actor-Critic算法性能。
摘要：
本文提出近端动作替换方法，解决离线强化学习中行为克隆导致的性能瓶颈问题，通过替换亚优动作提升Actor-Critic算法性能。
英文摘要：
Offline reinforcement learning (RL), which optimizes policies using a previously collected static dataset, is an important branch of RL. A popular and promising approach is to regularize actor-critic methods with behavior cloning (BC), which quickly yields realistic policies and mitigates bias from out-of-distribution actions, but it can impose an often-overlooked performance ceiling: when dataset actions are suboptimal, indiscriminate imitation structurally prevents the actor from fully exploiting better actions suggested by the value function, especially in later training when imitation is already dominant. We formally analyzed this limitation by investigating convergence properties of BC-regularized actor-critic optimization and verified it on a controlled continuous bandit task. To break this ceiling, we propose proximal action replacement (PAR), an easy-to-use plug-and-play training sample replacer. PAR substitutes suboptimal dataset actions with better actions generated by a stable target policy, guided by the action-value function's local ascent direction and bounded by value uncertainty to ensure training stability. PAR is compatible with multiple BC regularization paradigms. Extensive experiments across offline RL benchmarks show that PAR consistently improves performance, and approaches state-of-the-art results simply by being combined with the basic TD3+BC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
