CUDC：一种面向离线强化学习的、采用自适应时间距离的好奇心驱动无监督数据收集方法
CUDC: A Curiosity-Driven Unsupervised Data Collection Method with Adaptive Temporal Distances for Offline Reinforcement Learning
AI总结：
针对离线强化学习中数据收集受关注少、任务无关数据集收集难的问题，提出带自适应时间距离的好奇心驱动无监督数据收集方法CUDC，通过自适应可达性机制丰富特征、提升数据质量，在DeepMind控制套件任务中效率与性能优于现有方法。
摘要：
离线强化学习（RL）旨在从预先收集的数据集中学到有效策略。现有研究大多聚焦于开发复杂的学习算法，对改进数据收集流程的关注较少。此外，拓展单任务设置、收集能让智能体执行多个下游任务的任务无关数据集，更是一项挑战。本文提出好奇心驱动的无监督数据收集方法CUDC，通过自适应时间距离扩展特征空间，用于任务无关数据收集，最终提升多任务离线RL的学习效率与能力。为实现这一目标，CUDC估计从当前状态到达k步未来状态的概率，并自适应调整动力学模型需要预测的未来步数。借助这种自适应可达性机制，特征表示可实现多样化，智能体也能在好奇心驱动下自主导航以收集更高质量的数据。实验结果表明，在DeepMind控制套件的各类下游离线RL任务中，CUDC在效率与学习性能上均优于现有无监督方法。
英文摘要：
Offline reinforcement learning (RL) aims to learn an effective policy from a pre-collected dataset. Most existing works are to develop sophisticated learning algorithms, with less emphasis on improving the data collection process. Moreover, it is even challenging to extend the single-task setting and collect a task-agnostic dataset that allows an agent to perform multiple downstream tasks. In this paper, we propose a Curiosity-driven Unsupervised Data Collection (CUDC) method to expand feature space using adaptive temporal distances for task-agnostic data collection and ultimately improve learning efficiency and capabilities for multi-task offline RL. To achieve this, CUDC estimates the probability of the k-step future states being reachable from the current states, and adapts how many steps into the future that the dynamics model should predict. With this adaptive reachability mechanism in place, the feature representation can be diversified, and the agent can navigate itself to collect higher-quality data with curiosity. Empirically, CUDC surpasses existing unsupervised methods in efficiency and learning performance in various downstream offline RL tasks of the DeepMind control suite.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
