离线强化学习中基于策略的轨迹聚类
Policy-Based Trajectory Clustering in Offline Reinforcement Learning
AI总结：
该研究提出离线强化学习轨迹聚类新任务，构建基于KL散度的聚类目标，提出PG-Kmeans和CAAE两种方法，证明PG-Kmeans收敛性并指出最优解歧义问题，实验验证方法有效，为相关领域提供新框架。
摘要：
我们提出了一项全新任务：对离线强化学习（RL）数据集中的轨迹进行聚类，每个聚类中心代表生成其对应轨迹的策略。通过利用离线轨迹分布的KL散度与策略诱导分布混合模型之间的关联，我们构建了一个自然的聚类目标。为求解该目标，我们提出了策略引导K均值算法（Policy-Guided K-means, PG-Kmeans）和质心吸引自编码器（Centroid-Attracted Autoencoder, CAAE）。PG-Kmeans迭代训练行为克隆（behavior cloning, BC）策略，并基于策略生成概率分配轨迹；CAAE则类似VQ-VAE框架，通过引导轨迹的潜在表示向特定码本条目附近靠拢来实现聚类。理论上，我们证明了PG-Kmeans的有限步收敛性，并指出离线轨迹聚类的一项核心挑战：策略冲突导致的最优解固有歧义性，这会产生多个同等有效但结构不同的聚类结果。实验上，我们在广泛使用的D4RL数据集和自定义GridWorld环境上验证了方法的有效性。结果表明，PG-Kmeans和CAAE均能有效将轨迹划分为有意义的聚类，为基于策略的轨迹聚类提供了有前景的框架，在离线RL及更多领域具有广泛应用价值。
英文摘要：
We introduce a novel task of clustering trajectories from offline reinforcement learning (RL) datasets, where each cluster center represents the policy that generated its trajectories. By leveraging the connection between the KL-divergence of offline trajectory distributions and a mixture of policy-induced distributions, we formulate a natural clustering objective. To solve this, we propose Policy-Guided K-means (PG-Kmeans) and Centroid-Attracted Autoencoder (CAAE). PG-Kmeans iteratively trains behavior cloning (BC) policies and assigns trajectories based on policy generation probabilities, while CAAE resembles the VQ-VAE framework by guiding the latent representations of trajectories toward the vicinity of specific codebook entries to achieve clustering. Theoretically, we prove the finite-step convergence of PG-Kmeans and identify a key challenge in offline trajectory clustering: the inherent ambiguity of optimal solutions due to policy-induced conflicts, which can result in multiple equally valid but structurally distinct clusterings. Experimentally, we validate our methods on the widely used D4RL dataset and custom GridWorld environments. Our results show that both PG-Kmeans and CAAE effectively partition trajectories into meaningful clusters. They offer a promising framework for policy-based trajectory clustering, with broad applications in offline RL and beyond.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
