arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2202.01771cs.LGcs.CL
收藏

用于交互式决策的预训练语言模型

Pre-Trained Language Models for Interactive Decision-Making

MIT(麻省理工学院);Nvidia(英伟达);Caltech(加州理工学院);Google Brain(谷歌大脑);UT Austin(德克萨斯大学奥斯汀分校)
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, An… 展开作者

Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, Yuke Zhu

摘要:

语言模型(LM)预训练在许多语言处理任务中是有用的。但是,预训练的LM是否可以进一步用于更一般的机器学习问题?我们提出了一种方法,使用LM脚手架学习和一般顺序决策问题的推广。在这种方法中,目标和观察被表示为一系列嵌入,并且用预先训练的LM初始化的策略网络预测下一个动作。我们证明了这个框架,使有效的组合推广在不同的环境和监督方式。我们首先假设可以访问一组专家演示,并表明使用LM初始化策略并通过行为克隆对其进行微调可以将VirtualHome环境中的任务完成率提高43.6%。接下来,我们集成了一个主动的数据收集过程,在这个过程中,代理人迭代地与环境交互,用新的目标重新标记过去的“失败”经历,并在自我监督的循环中更新他们的策略。主动数据收集进一步提高了组合泛化能力,比最佳基线高出25.1%。最后,我们解释这些结果,调查三个可能的因素的有效性LM为基础的政策。我们发现,顺序输入表示(与固定维的特征向量)和基于LM的权重初始化都是很重要的泛化。然而,令人惊讶的是,策略输入编码的格式(例如,作为自然语言字符串与任意顺序编码)几乎没有影响。总之,这些结果表明,语言建模诱导的表征不仅对语言建模有用,而且对目标和计划也有用;这些表征可以帮助学习和泛化,甚至在语言处理之外。

英文摘要:

Language model (LM) pre-training is useful in many language processing tasks. But can pre-trained LMs be further leveraged for more general machine learning problems? We propose an approach for using LMs to scaffold learning and generalization in general sequential decision-making problems. In this approach, goals and observations are represented as a sequence of embeddings, and a policy network initialized with a pre-trained LM predicts the next action. We demonstrate that this framework enables effective combinatorial generalization across different environments and supervisory modalities. We begin by assuming access to a set of expert demonstrations, and show that initializing policies with LMs and fine-tuning them via behavior cloning improves task completion rates by 43.6% in the VirtualHome environment. Next, we integrate an active data gathering procedure in which agents iteratively interact with the environment, relabel past "failed" experiences with new goals, and update their policies in a self-supervised loop. Active data gathering further improves combinatorial generalization, outperforming the best baseline by 25.1%. Finally, we explain these results by investigating three possible factors underlying the effectiveness of the LM-based policy. We find that sequential input representations (vs. fixed-dimensional feature vectors) and LM-based weight initialization are both important for generalization. Surprisingly, however, the format of the policy inputs encoding (e.g. as a natural language string vs. an arbitrary sequential encoding) has little influence. Together, these results suggest that language modeling induces representations that are useful for modeling not just language, but also goals and plans; these representations can aid learning and generalization even outside of language processing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑