PROTO：迭代策略正则化的离线到在线强化学习
PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning
AI总结：
PROTO 通过在标准 RL 目标中加入迭代演化的正则化项，以信赖域式更新逐步放宽约束，实现稳定且高效的离线到在线强化学习微调，并可与任意离线预训练方法对接。
摘要：
离线到在线强化学习（RL）通过结合离线预训练与在线微调的优势，有望提升样本效率和策略性能。然而，现有方法虽然有效，却存在性能次优、适应性有限以及计算效率不理想的问题。我们提出了一种新颖的框架 PROTO，它通过在标准 RL 目标上增加一个迭代演化的正则化项来克服上述局限。PROTO 执行信赖域风格的更新，通过逐步演化正则化项以放宽约束强度，从而在初始微调阶段保持稳定，并最终获得最优性能。只需调整少量代码，PROTO 就能将任意离线策略预训练与标准的离线策略 RL 微调相连接，形成强大的离线到在线 RL 通路，从而对多种方法具有很强的适应性。PROTO 简单而优雅，仅引入极少的额外计算量，并支持高效的在线微调。大量实验表明，PROTO 相比 SOTA 基线取得了更优的性能，提供了一个适应性强且高效的离线到在线 RL 框架。
英文摘要：
Offline-to-online reinforcement learning (RL), by combining the benefits of offline pretraining and online finetuning, promises enhanced sample efficiency and policy performance. However, existing methods, effective as they are, suffer from suboptimal performance, limited adaptability, and unsatisfactory computational efficiency. We propose a novel framework, PROTO, which overcomes the aforementioned limitations by augmenting the standard RL objective with an iteratively evolving regularization term. Performing a trust-region-style update, PROTO yields stable initial finetuning and optimal final performance by gradually evolving the regularization term to relax the constraint strength. By adjusting only a few lines of code, PROTO can bridge any offline policy pretraining and standard off-policy RL finetuning to form a powerful offline-to-online RL pathway, birthing great adaptability to diverse methods. Simple yet elegant, PROTO imposes minimal additional computation and enables highly efficient online finetuning. Extensive experiments demonstrate that PROTO achieves superior performance over SOTA baselines, offering an adaptable and efficient offline-to-online RL framework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
