arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.09115cs.CL
收藏

长程语言模型真的使用长程上下文吗?

Do Long-Range Language Models Actually Use Long-Range Context?

University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);Intuit AI(Intuit AI(Intuit人工智能部门))

Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, Mohit Iyyer

摘要:

语言模型通常在短的、截断的输入序列上训练,这限制了它们使用长范围上下文中存在的话语级信息来改进其预测的能力。最近的努力,以提高自我注意力的效率,导致了扩散的远程Transformer语言模型,它可以处理更长的序列比过去的模型。然而,这些模型利用长期背景的方式仍不清楚。在本文中,我们对两个远程Transformer语言模型(包括\emph{Routing Transformer},它在PG-19长序列LM基准数据集上实现了最先进的困惑)进行了细粒度分析,这些模型接受高达8 K的输入序列。我们的研究结果表明,提供长期背景(即,超出先前的2K令牌)到这些模型仅改进了它们对一小组令牌的预测(例如,可以从远距离上下文复制的那些),并且对于同步级预测任务根本没有帮助。最后,我们发现PG-19包含各种不同的文档类型和领域,并且长期上下文对文学小说(而不是教科书或杂志)帮助最大。

英文摘要:

Language models are generally trained on short, truncated input sequences, which limits their ability to use discourse-level information present in long-range context to improve their predictions. Recent efforts to improve the efficiency of self-attention have led to a proliferation of long-range Transformer language models, which can process much longer sequences than models of the past. However, the ways in which such models take advantage of the long-range context remain unclear. In this paper, we perform a fine-grained analysis of two long-range Transformer language models (including the \emph{Routing Transformer}, which achieves state-of-the-art perplexity on the PG-19 long-sequence LM benchmark dataset) that accept input sequences of up to 8K tokens. Our results reveal that providing long-range context (i.e., beyond the previous 2K tokens) to these models only improves their predictions on a small set of tokens (e.g., those that can be copied from the distant context) and does not help at all for sentence-level prediction tasks. Finally, we discover that PG-19 contains a variety of different document types and domains, and that long-range context helps most for literary novels (as opposed to textbooks or magazines).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑