行为自适应Q学习：离线到在线强化学习的统一框架
Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
AI总结：
本文提出行为自适应Q学习（BAQ），通过离线数据构建隐式行为模型并在在线微调中提供行为一致性约束，随在线经验积累逐步放宽约束，从而稳定离线到在线强化学习的过渡并提升最终性能。
摘要：
离线强化学习（RL）能够在没有在线交互的情况下从固定数据中进行训练，但由于分布偏移以及对未见过的状态-动作对的价值估计不可靠，离线学习到的策略在部署到动态环境中时往往表现不佳。我们提出了行为自适应Q学习（BAQ），这是一个旨在实现从离线RL到在线RL平滑且可靠过渡的框架。其核心思想是利用从离线数据中推导出的隐式行为模型，在在线微调过程中提供行为一致性信号。BAQ采用了一个双目标损失函数，该损失函数（i）在不确定性较高时使在线策略向离线行为对齐，并且（ii）随着积累到更可信的在线经验，逐步放宽这一约束。这种自适应机制减少了来自分布外估计的误差传播，稳定了早期在线更新，并加速了对新场景的适应。在标准基准测试中，BAQ一致地优于以往的离线到在线RL方法，实现了更快的恢复、更强的鲁棒性以及更高的整体性能。我们的结果表明，隐式行为自适应是一种具有原则性且实用的解决方案，可用于可靠的现实世界策略部署。
英文摘要：
Offline reinforcement learning (RL) enables training from fixed data without online interaction, but policies learned offline often struggle when deployed in dynamic environments due to distributional shift and unreliable value estimates on unseen state-action pairs. We introduce Behavior-Adaptive Q-Learning (BAQ), a framework designed to enable a smooth and reliable transition from offline to online RL. The key idea is to leverage an implicit behavioral model derived from offline data to provide a behavior-consistency signal during online fine-tuning. BAQ incorporates a dual-objective loss that (i) aligns the online policy toward the offline behavior when uncertainty is high, and (ii) gradually relaxes this constraint as more confident online experience is accumulated. This adaptive mechanism reduces error propagation from out-of-distribution estimates, stabilizes early online updates, and accelerates adaptation to new scenarios. Across standard benchmarks, BAQ consistently outperforms prior offline-to-online RL approaches, achieving faster recovery, improved robustness, and higher overall performance. Our results demonstrate that implicit behavior adaptation is a principled and practical solution for reliable real-world policy deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
