arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.09550cs.LGcs.RO
收藏

用于改进离线策略学习的数据集聚类

Dataset Clustering for Improved Offline Policy Learning

University College Dublin(都柏林大学学院);Dublin City University(都柏林城市大学);Insight SFI Research Centre for Data Analytics(Insight SFI数据分析研究中心)

Qiang Wang, Yixin Deng, Francisco Roldan Sanchez, Keru Wang, Kevin McGuinness, Noel O'Connor, Stephen J. Redmond

AI总结:

本文提出一种行为感知深度聚类方法,将多行为数据集自适应划分为单行为子集,以提升离线策略学习性能,在连续控制任务中聚类平均ARI达0.987。

摘要:

离线策略学习旨在从先前收集的数据集中发现决策策略,而无需与环境进行额外的在线交互。由于训练数据集是固定的,其质量成为学习策略性能的关键决定因素。本文研究了一种我们称之为多行为的数据集特征,表示该数据集是使用表现出不同行为的多个策略收集的。相反,单行为数据集仅使用一个策略收集。我们观察到,从单行为数据集中学习的策略通常优于从多行为数据集中学习的策略,尽管单行为数据集具有更少的样本和更少的多样性。因此,我们提出了一种行为感知深度聚类方法,将多行为数据集划分为多个单行为子集,从而有利于下游策略学习。我们的方法灵活且有效;它能够自适应地估计簇的数量,同时展现出高聚类准确率,在各种连续控制任务数据集上实现了0.987的平均调整兰德指数。最后,我们展示了使用数据集聚类改进策略学习的示例,并讨论了我们的方法可能有益于离线策略学习社区的几种潜在场景。

英文摘要:

Offline policy learning aims to discover decision-making policies from previously-collected datasets without additional online interactions with the environment. As the training dataset is fixed, its quality becomes a crucial determining factor in the performance of the learned policy. This paper studies a dataset characteristic that we refer to as multi-behavior, indicating that the dataset is collected using multiple policies that exhibit distinct behaviors. In contrast, a uni-behavior dataset would be collected solely using one policy. We observed that policies learned from a uni-behavior dataset typically outperform those learned from multi-behavior datasets, despite the uni-behavior dataset having fewer examples and less diversity. Therefore, we propose a behavior-aware deep clustering approach that partitions multi-behavior datasets into several uni-behavior subsets, thereby benefiting downstream policy learning. Our approach is flexible and effective; it can adaptively estimate the number of clusters while demonstrating high clustering accuracy, achieving an average Adjusted Rand Index of 0.987 across various continuous control task datasets. Finally, we present improved policy learning examples using dataset clustering and discuss several potential scenarios where our approach might benefit the offline policy learning community.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑