arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00629stat.MLcs.AIcs.LG
收藏

无需动作的离线到在线强化学习 via 离散化状态策略

Action-Free Offline-to-Online RL via Discretised State Policies

University of Warwick(华威大学);Nanyang Technological University(南洋理工大学);The Alan Turing Institute(艾伦·图灵研究所)

Natinael Solomon Neggatu, Jeremie Houssineau, Giovanni Montana

AI总结:

本文提出无需动作的离线到在线强化学习方法,通过离散化状态策略提升高维问题的收敛速度和渐近性能。

摘要:

现有大多数离线强化学习方法假定数据集中存在动作标签,但在许多实际场景中,由于隐私、存储或传感器限制,动作可能缺失。我们正式定义了无需动作的离线到在线强化学习设置,其中智能体必须从仅包含(s,r,s')元组的数据集中学习,并在在线交互中利用此知识。为解决这一挑战,我们提出学习状态策略,推荐有益的下一步状态转移,而不是动作。我们的贡献是双方面的。首先,我们引入了一种简单但新颖的状态离散化转换,并提出了一种基于价值的算法,即Offline State-Only DecQN(算法),该算法旨在从无动作数据中预训练状态策略。算法集成了该转换,能够高效扩展到高维问题,同时避免与连续状态预测相关的不稳定性过拟合问题。其次,我们提出了一种新的机制,用于引导在线学习,利用这些预训练的状态策略来加速在线智能体的学习。这些组件共同建立了一个可扩展且实用的框架,用于利用无动作数据来加速在线强化学习。在多样化的基准测试中,我们的方法在收敛速度和渐近性能方面有所提升,分析表明离散化和正则化对于其有效性至关重要。

英文摘要:

Most existing offline RL methods presume the availability of action labels within the dataset, but in many practical scenarios, actions may be missing due to privacy, storage, or sensor limitations. We formalise the setting of action-free offline-to-online RL, where agents must learn from datasets consisting solely of $(s,r,s')$ tuples and later leverage this knowledge during online interaction. To address this challenge, we propose learning state policies that recommend desirable next-state transitions rather than actions. Our contributions are twofold. First, we introduce a simple yet novel state discretisation transformation and propose Offline State-Only DecQN (\algo), a value-based algorithm designed to pre-train state policies from action-free data. \algo{} integrates the transformation to scale efficiently to high-dimensional problems while avoiding instability and overfitting associated with continuous state prediction. Second, we propose a novel mechanism for guided online learning that leverages these pre-trained state policies to accelerate the learning of online agents. Together, these components establish a scalable and practical framework for leveraging action-free datasets to accelerate online RL. Empirical results across diverse benchmarks demonstrate that our approach improves convergence speed and asymptotic performance, while analyses reveal that discretisation and regularisation are critical to its effectiveness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑