arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.05290cs.LGcs.AI
收藏

Transformer世界模型能提供更好的策略梯度吗?

Do Transformer World Models Give Better Policy Gradients?

Mila(米拉研究所);Université de Montréal(蒙特利尔大学)

Michel Ma, Tianwei Ni, Clement Gehring, Pierluca D'Oro, Pierre-Luc Bacon

AI总结:

针对Transformer世界模型在长视野下产生迂回梯度路径的问题,提出Actions World Models (AWMs)提供更直接的梯度传播路径,生成更易导航的优化景观,从而在长视野任务中产生更优策略。

摘要:

强化学习的一种自然方法是通过展开神经网络世界模型来预测未来奖励,并通过生成的计算图进行反向传播以学习策略。然而,由于典型的世界模型会导致难以优化的损失景观,这种方法在长视野下通常变得不切实际。已知Transformer能在长视野上高效传播梯度:它们能解决这一问题吗?令人惊讶的是,我们表明常用的Transformer世界模型会产生迂回的梯度路径,这可能对长程策略梯度不利。为应对这一挑战,我们提出了一类世界模型,称为Actions World Models (AWMs),旨在为梯度传播提供更直接的路径。我们将此类AWMs集成到一个策略梯度框架中,该框架强调了网络架构与其内在代表的策略梯度更新之间的关系。我们证明,与模拟器本身产生的优化景观相比,AWMs能生成更易于导航的优化景观。这一特性使得Transformer AWM在现实的长视野任务中能产生比竞争性基线更好的策略。

英文摘要:

A natural approach for reinforcement learning is to predict future rewards by unrolling a neural network world model, and to backpropagate through the resulting computational graph to learn a policy. However, this method often becomes impractical for long horizons since typical world models induce hard-to-optimize loss landscapes. Transformers are known to efficiently propagate gradients over long horizons: could they be the solution to this problem? Surprisingly, we show that commonly-used transformer world models produce circuitous gradient paths, which can be detrimental to long-range policy gradients. To tackle this challenge, we propose a class of world models called Actions World Models (AWMs), designed to provide more direct routes for gradient propagation. We integrate such AWMs into a policy gradient framework that underscores the relationship between network architectures and the policy gradient updates they inherently represent. We demonstrate that AWMs can generate optimization landscapes that are easier to navigate even when compared to those from the simulator itself. This property allows transformer AWMs to produce better policies than competitive baselines in realistic long-horizon tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑