arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2104.05500cs.CLcs.AIcs.LG
收藏

用于归纳世界状态表示的更新器-提取器架构

Updater-Extractor Architecture for Inductive World State Representations

University of California, Irvine(加州大学尔湾分校);K-Sky Limited(K-Sky有限公司)

Arseny Moskvichev, James A. Liu

摘要:

开发NLP模型传统上包括两个阶段-培训和应用。在训练之后(在应用时)获取的信息的保留在架构上受到模型的上下文窗口的大小(在Transformers的情况下)或与长序列相关的实际困难(在RNN的情况下)的限制。在本文中,我们提出了一种新的基于transformer的Updater-Extractor架构和一个训练过程,可以处理任意长度的序列,并根据语言输入完善其对世界的知识。我们显式地训练模型,将传入的信息纳入其世界状态表示,获得强大的归纳概括和处理极长范围的依赖关系的能力。我们证明了一个引理,为我们的方法提供了理论基础。结果还提供了对使用截断反向传播时间变体(例如Transformer XL)训练的模型的成功和失败模式的洞察。从经验上讲,我们研究了模型在三个不同任务上的性能,证明了它的承诺。这个预印本仍然是一个正在进行的工作。目前,我们专注于易于解释的任务,将所提出的思想应用于未来的实际NLP应用。

英文摘要:

Developing NLP models traditionally involves two stages - training and application. Retention of information acquired after training (at application time) is architecturally limited by the size of the model's context window (in the case of transformers), or by the practical difficulties associated with long sequences (in the case of RNNs). In this paper, we propose a novel transformer-based Updater-Extractor architecture and a training procedure that can work with sequences of arbitrary length and refine its knowledge about the world based on linguistic inputs. We explicitly train the model to incorporate incoming information into its world state representation, obtaining strong inductive generalization and the ability to handle extremely long-range dependencies. We prove a lemma that provides a theoretical basis for our approach. The result also provides insight into success and failure modes of models trained with variants of Truncated Back-Propagation Through Time (such as Transformer XL). Empirically, we investigate the model performance on three different tasks, demonstrating its promise. This preprint is still a work in progress. At present, we focused on easily interpretable tasks, leaving the application of the proposed ideas to practical NLP applications for the future.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑