基于信念的离线强化学习用于延迟鲁棒策略优化
Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization
AI总结：
DT-CORL通过基于信念的变压器预测器和约束离线强化学习框架，实现延迟鲁棒的策略优化，提升仿真到现实的迁移性能。
摘要：
离线到在线部署的强化学习代理必须弥合两个差距：（1）仿真到现实的差距，其中现实系统添加延迟和其他仿真中没有的不完美之处；（2）交互差距，其中纯离线训练的策略在在线执行时会遇到分布外状态，因为收集新交互数据成本高昂或风险大。因此，代理必须从静态、无延迟的数据集泛化到动态、延迟敏感的环境。标准离线强化学习从无延迟日志中学习，但在部署时必须在存在延迟的情况下行动，这会破坏马尔可夫假设并损害性能。我们引入DT-CORL（延迟转换器信念策略约束离线强化学习），一种专门设计应对部署时延迟动态的离线强化学习框架。DT-CORL（i）能够生成延迟鲁棒的动作，即使在训练过程中从未见过延迟观察；（ii）比朴素的历史增强基线明显更高效。在D4RL基准测试中，使用几种延迟设置进行实验，结果表明DT-CORL在性能上始终优于历史增强和纯信念方法，缩小了仿真到现实的延迟差距，同时保持了数据效率。
英文摘要：
Offline-to-online deployment of reinforcement-learning (RL) agents must bridge two gaps: (1) the sim-to-real gap, where real systems add latency and other imperfections not present in simulation, and (2) the interaction gap, where policies trained purely offline face out-of-distribution states during online execution because gathering new interaction data is costly or risky. Agents therefore have to generalize from static, delay-free datasets to dynamic, delay-prone environments. Standard offline RL learns from delay-free logs yet must act under delays that break the Markov assumption and hurt performance. We introduce DT-CORL (Delay-Transformer belief policy Constrained Offline RL), an offline-RL framework built to cope with delayed dynamics at deployment. DT-CORL (i) produces delay-robust actions with a transformer-based belief predictor even though it never sees delayed observations during training, and (ii) is markedly more sample-efficient than naïve history-augmentation baselines. Experiments on D4RL benchmarks with several delay settings show that DT-CORL consistently outperforms both history-augmentation and vanilla belief-based methods, narrowing the sim-to-real latency gap while preserving data efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
