随机丢帧下的判决Transformer
Decision Transformer under Random Frame Dropping
摘要：
在现实世界中使用深度强化学习~（DRL）远程控制代理还有待实现。一个关键的垫脚石是设计RL算法，这些算法在面对来自损坏的通信或故障传感器的丢失信息时具有鲁棒性。典型的强化学习方法通常需要大量的在线交互数据，这些数据在现实世界中收集起来既昂贵又不安全。此外，当应用于帧丢弃场景时，即使具有中等的丢弃率，它们也不能令人满意地执行。为了解决这些问题，我们提出了决策Transformer下随机帧丢弃~（DeFog），离线RL算法，使代理在帧丢弃的情况下，没有在线交互的鲁棒性。DeFog首先随机屏蔽离线数据集中的数据，并显式添加帧丢弃的时间跨度作为输入。在此之后，对同一离线数据集进行微调，并使用更高的掩码率，将进一步提高性能。实证结果表明，DeFog在严重的帧丢失率（如90%）下优于强基线，同时在常规MuJoCo控制基准和Atari环境中的非帧丢失条件下保持类似的回报。我们的方法提供了一个强大的和可部署的解决方案，在现实世界的环境中控制代理有限或不可靠的数据。
英文摘要：
Controlling agents remotely with deep reinforcement learning~(DRL) in the real world is yet to come. One crucial stepping stone is to devise RL algorithms that are robust in the face of dropped information from corrupted communication or malfunctioning sensors. Typical RL methods usually require considerable online interaction data that are costly and unsafe to collect in the real world. Furthermore, when applying to the frame dropping scenarios, they perform unsatisfactorily even with moderate drop rates. To address these issues, we propose Decision Transformer under Random Frame Dropping~(DeFog), an offline RL algorithm that enables agents to act robustly in frame dropping scenarios without online interaction. DeFog first randomly masks out data in the offline datasets and explicitly adds the time span of frame dropping as inputs. After that, a finetuning stage on the same offline dataset with a higher mask rate would further boost the performance. Empirical results show that DeFog outperforms strong baselines under severe frame drop rates like 90\%, while maintaining similar returns under non-frame-dropping conditions in the regular MuJoCo control benchmarks and the Atari environments. Our approach offers a robust and deployable solution for controlling agents in real-world environments with limited or unreliable data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
