COOPO：循环离线-在线策略优化算法
COOPO: Cyclic Offline-Online Policy Optimization Algorithm
AI总结：
本文提出COOPO算法，通过循环离线训练和在线微调来解决离线强化学习的分布偏移和性能受限问题，以及在线强化学习的环境交互成本高问题，通过周期性回归离线训练减少遗忘和漂移，提升样本效率和性能。
摘要：
离线强化学习由于静态数据集的限制，在面对分布偏移和受限性能方面存在困难，而在线强化学习则需要大量的环境交互。最近出现的混合离线-在线方法连接了这两个领域，但存在转换过程中的分布漂移和对离线知识的灾难性遗忘问题。我们引入COOPO（循环离线-在线策略优化），一种通用框架，通过反复循环在受限的离线训练和在线微调之间进行。每个循环首先通过KL-正则化的优势加权离线更新将策略锚定到数据集，以最小化分布偏移，然后使用任何策略优化方法在线微调以实现稳定的探索。关键的是，定期返回离线训练可以消除遗忘和漂移，同时最大化数据集的再利用。循环行为还帮助减少在线环境交互。理论上，COOPO在样本效率上优于纯在线RL，满足标准覆盖假设下保证单调改进。广泛的D4RL基准测试显示，COOPO在减少在线交互的同时提高最终回报，保持在不同离线算法和在线优化器中的鲁棒性。这种循环协同为自适应RL设定了新的效率和性能标准。
英文摘要：
Offline reinforcement learning struggles with distributional shift and constrained performance due to static dataset limitations, while online RL demands prohibitive environment interactions. The recent advent of hybrid offline-to-online methods bridges these domains but suffers from distribution drift during transitions and catastrophic forgetting of offline knowledge. We introduce COOPO (Cyclic Offline-Online Policy Optimization), a generalized framework that repeatedly cycles between constrained offline training and online fine-tuning. Each cycle first anchors the policy to the dataset via KL-regularized advantage-weighted offline updates to minimize distributional shift and then fine-tunes it online using any policy optimization for stable exploration. Crucially, periodically returning to offline training eliminates forgetting and drift while maximizing dataset reuse. The cyclic behavior also helps reduce the online environment interactions. Theoretically, COOPO achieves better online sample efficiency, surpassing pure online RL, with guaranteed monotonic improvement under standard coverage assumptions. Extensive D4RL benchmarks demonstrate COOPO reduces online interactions versus state-of-the-art hybrids while improving final returns, maintaining robustness across diverse offline algorithms and online optimizers. This looped synergy sets new efficiency and performance standards for adaptive RL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
