非平稳环境下的离线强化学习预测
Forecasting in Offline Reinforcement Learning for Non-stationary Environments
AI总结：
非平稳环境下的离线强化学习预测框架FORL，通过结合条件扩散和零样本时间序列模型，提升智能体在非平稳环境中的鲁棒性与性能。
摘要：
离线强化学习（RL）提供了一种有前景的途径，用于从预收集的数据集中训练策略，当获取额外交互数据不可行时。然而，现有离线RL方法通常假设平稳性或仅在测试时考虑合成扰动，这些假设在现实世界场景中往往失效，现实世界场景以突然的时间变化偏移为特征。这些偏移可能导致部分可观测性，导致智能体错误感知其真实状态并降低性能。为克服这一挑战，我们引入了非平稳离线RL中的预测（FORL）框架，该框架统一了（i）基于条件扩散的候选状态生成，该生成器无需假设任何特定的未来非平稳性模式，以及（ii）零样本时间序列基础模型。FORL针对容易出现意外、可能非马尔可夫偏移的环境，要求从每个回合的开始就具备稳健的智能体性能。在离线RL基准测试中，通过加入现实世界时间序列数据以模拟现实的非平稳性，实验证明FORL在与竞争基线相比时性能持续提升。通过将零样本预测与智能体的经验相结合，我们旨在弥合离线RL与现实世界非平稳环境复杂性的差距。
英文摘要：
Offline Reinforcement Learning (RL) provides a promising avenue for training policies from pre-collected datasets when gathering additional interaction data is infeasible. However, existing offline RL methods often assume stationarity or only consider synthetic perturbations at test time, assumptions that often fail in real-world scenarios characterized by abrupt, time-varying offsets. These offsets can lead to partial observability, causing agents to misperceive their true state and degrade performance. To overcome this challenge, we introduce Forecasting in Non-stationary Offline RL (FORL), a framework that unifies (i) conditional diffusion-based candidate state generation, trained without presupposing any specific pattern of future non-stationarity, and (ii) zero-shot time-series foundation models. FORL targets environments prone to unexpected, potentially non-Markovian offsets, requiring robust agent performance from the onset of each episode. Empirical evaluations on offline RL benchmarks, augmented with real-world time-series data to simulate realistic non-stationarity, demonstrate that FORL consistently improves performance compared to competitive baselines. By integrating zero-shot forecasting with the agent's experience, we aim to bridge the gap between offline RL and the complexities of real-world, non-stationary environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
