arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.15909cs.LGcs.AI
收藏

数据受限下的离线元强化学习通用任务表示学习

Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations

Nanjing University(南京大学)

Renzhe Zhou, Chen-Xiao Gao, Zongzhang Zhang, Yang Yu

AI总结:

本文提出GENTLE算法,通过任务自编码器仅重构状态转移和奖励,并构建伪转移对齐数据分布,解决离线元强化学习中训练任务和行为多样性有限的问题,显著提升泛化能力。

摘要:

泛化能力和样本效率一直是强化学习长期存在的问题,因此离线元强化学习(OMRL)领域因其能够利用静态且有限的离线数据解决广泛问题的潜力而受到越来越多的关注。现有的OMRL方法通常假设有足够的训练任务和数据覆盖度,以便应用对比学习来提取任务表示。然而,这些假设在多个现实应用场景中并不成立,从而削弱了表示的泛化能力。在本文中,我们考虑两种数据受限情况下的OMRL:训练任务数量有限和行为多样性有限,并提出了一种名为GENTLE的新算法,用于在数据受限的情况下学习可泛化的任务表示。GENTLE采用任务自编码器(TAE),这是一种编码器-解码器架构,用于提取任务的特征。与现有方法不同,TAE仅通过状态转移和奖励的重构进行优化,这捕捉了任务模型的生成结构,并在训练任务有限时产生可泛化的表示。为了减轻行为多样性有限的影响,我们持续构建伪转移,以使训练TAE所用的数据分布与测试时遇到的数据分布对齐。实验上,GENTLE在分布内任务和分布外任务上,在给定上下文协议和一次性协议下,均显著优于现有的OMRL方法。

英文摘要:

Generalization and sample efficiency have been long-standing issues concerning reinforcement learning, and thus the field of Offline Meta-Reinforcement Learning~(OMRL) has gained increasing attention due to its potential of solving a wide range of problems with static and limited offline data. Existing OMRL methods often assume sufficient training tasks and data coverage to apply contrastive learning to extract task representations. However, such assumptions are not applicable in several real-world applications and thus undermine the generalization ability of the representations. In this paper, we consider OMRL with two types of data limitations: limited training tasks and limited behavior diversity and propose a novel algorithm called GENTLE for learning generalizable task representations in the face of data limitations. GENTLE employs Task Auto-Encoder~(TAE), which is an encoder-decoder architecture to extract the characteristics of the tasks. Unlike existing methods, TAE is optimized solely by reconstruction of the state transition and reward, which captures the generative structure of the task models and produces generalizable representations when training tasks are limited. To alleviate the effect of limited behavior diversity, we consistently construct pseudo-transitions to align the data distribution used to train TAE with the data distribution encountered during testing. Empirically, GENTLE significantly outperforms existing OMRL methods on both in-distribution tasks and out-of-distribution tasks across both the given-context protocol and the one-shot protocol.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑