arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.04590cs.LGcs.AI
收藏

通过压缩学习选项

Learning Options via Compression

Carnegie Mellon University(卡内基梅隆大学);Stanford University(斯坦福大学)

Yiding Jiang, Evan Zheran Liu, Benjamin Eysenbach, Zico Kolter, Chelsea Finn

摘要:

在多任务强化学习中,识别某些任务的解决方案中的统计错误可以加速新任务的学习。技能学习通过将预先收集的经验分解为一系列技能,提供了一种识别这些技能的方法。一种流行的技能学习方法是使用潜变量模型最大化预先收集的经验的可能性,其中潜变量表示技能。然而,通常有许多解决方案,最大化的可能性同样好,包括退化的解决方案。为了解决这个不足,我们提出了一个新的目标,结合了最大似然目标与惩罚的描述长度的技能。这种惩罚激励技能从经验中最大限度地提取共同结构。从经验上讲,我们的目标学习技能,解决下游任务在更少的样本相比,技能只从最大化的可能性。此外,虽然离线多任务设置中的大多数先前的工作集中在具有低维观测的任务上,但我们的目标可以扩展到具有高维图像观测的具有挑战性的任务。

英文摘要:

Identifying statistical regularities in solutions to some tasks in multi-task reinforcement learning can accelerate the learning of new tasks. Skill learning offers one way of identifying these regularities by decomposing pre-collected experiences into a sequence of skills. A popular approach to skill learning is maximizing the likelihood of the pre-collected experience with latent variable models, where the latent variables represent the skills. However, there are often many solutions that maximize the likelihood equally well, including degenerate solutions. To address this underspecification, we propose a new objective that combines the maximum likelihood objective with a penalty on the description length of the skills. This penalty incentivizes the skills to maximally extract common structures from the experiences. Empirically, our objective learns skills that solve downstream tasks in fewer samples compared to skills learned from only maximizing likelihood. Further, while most prior works in the offline multi-task setting focus on tasks with low-dimensional observations, our objective can scale to challenging tasks with high-dimensional image observations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑