arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.16554cs.LG
收藏

从后见之明经验链中涌现的智能体 Transformer

Emergent Agentic Transformer from Chain of Hindsight Experience

University of California, Berkeley(加州大学伯克利分校)

Hao Liu, Pieter Abbeel

AI总结:

本文提出智能体 Transformer(AT),通过后见之明链重新标记次优轨迹并训练自回归模型,使基于 Transformer 的策略能直接组合多个次优经验,在 D4RL 和 ExoRL 上首次与 TD 和模仿学习方法相竞争,并展现随模型规模扩大而持续提升的趋势。

摘要:

由多样化数据和模型规模驱动的大型 Transformer 模型已在自然语言建模和计算机视觉领域占据主导地位,并推动了多个 AI 领域的前沿发展。在强化学习(RL)中,尽管人们在基于 Transformer 的策略上付出了许多努力,但一个关键限制是,当前的基于 Transformer 的策略无法通过直接组合来自多个次优试验的信息进行学习。在本工作中,我们使用最近提出的后见之明链来重新标记经验,以解决这一问题:我们在一个按总奖励升序排序的轨迹经验序列上训练 Transformer。我们的方法包括将每条轨迹的目标回报重新标记为该轨迹序列中的最大总奖励,并训练一个自回归模型,以过去的状态、动作、奖励、目标回报和任务完成标记为条件来预测动作,所得模型即智能体 Transformer(AT),它能够在训练和测试时学会自我改进。正如我们在 D4RL 和 ExoRL 基准上所示,据我们所知,这是首次一个简单的基于 Transformer 的模型能够与基于时序差分和基于模仿学习的方法相竞争,即使是从次优数据中学习也是如此。我们的智能体 Transformer 还展现出有前景的扩展趋势,即更大的模型能够持续改善结果。

英文摘要:

Large transformer models powered by diverse data and model scale have dominated natural language modeling and computer vision and pushed the frontier of multiple AI areas. In reinforcement learning (RL), despite many efforts into transformer-based policies, a key limitation, however, is that current transformer-based policies cannot learn by directly combining information from multiple sub-optimal trials. In this work, we address this issue using recently proposed chain of hindsight to relabel experience, where we train a transformer on a sequence of trajectory experience ascending sorted according to their total rewards. Our method consists of relabelling target return of each trajectory to the maximum total reward among in sequence of trajectories and training an autoregressive model to predict actions conditioning on past states, actions, rewards, target returns, and task completion tokens, the resulting model, Agentic Transformer (AT), can learn to improve upon itself both at training and test time. As we show on D4RL and ExoRL benchmarks, to the best our knowledge, this is the first time that a simple transformer-based model performs competitively with both temporal-difference and imitation-learning-based approaches, even from sub-optimal data. Our Agentic Transformer also shows a promising scaling trend that bigger models consistently improve results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑