AI 中文总结
本文将校准模型首次应用于市政水处理厂的真实RL场景，评估其在高维非平稳传感器数据上的表现，为离线动力学模型支持真实RL部署提供概念验证。
AI 中文摘要
将强化学习（RL）部署到真实系统的一个关键障碍是超参数选择，尤其是在模拟器不可用且在线实验成本高昂的情况下。此前研究提出了基于离线数据训练的校准模型，用于近似环境动力学并实现离线超参数选择，但这些方法迄今仅在简单模拟环境中得到评估。本文首次将校准模型应用于真实工业场景：市政水处理厂。我们评估了多种校准模型方法，包括采用拉普拉斯距离度量的k近邻模型，用于高维非平稳传感器数据的nexting预测任务。结果表明，这些模型可生成逼真的长时域滚动输出，并恢复有意义的超参数敏感性趋势。我们进一步研究了校准模型在长达一年的数据集上的扩展性、如何支持预训练智能体的微调学习率选择，以及在分布偏移下的鲁棒性。总体而言，我们的发现为使用离线动力学模型支持RL在真实环境中的部署提供了概念验证，同时也指出了未来工作需解决的重要实际挑战。
英文摘要
A key obstacle to deploying reinforcement learning in real-world systems is hyperparameter selection, particularly when simulators are unavailable and online experimentation is costly. Prior work has proposed calibration models trained on offline data to approximate environment dynamics and enable offline hyperparameter selection, but these methods have so far been evaluated only in simple simulated settings. In this paper, we present the first application of calibration models in a real-world industrial setting: a municipal water treatment plant. We evaluate several calibration model approaches, including a k-nearest neighbors model with a Laplacian distance metric, on high-dimensional, non-stationary sensor data for nexting prediction tasks. Our results show that these models can generate realistic long-horizon rollouts and recover meaningful hyperparameter sensitivity trends. We further examine how calibration models scale to year-long datasets, how they support the selection of fine-tuning learning rates for pre-trained agents, and how robust they are under distribution shift. Overall, our findings provide a proof of concept for using offline dynamics models to support RL deployment in real-world environments, while highlighting important practical challenges for future work.
CommentsAccepted to the 2026 Reinforcement Learning Conference (RLC)
