神经语言建模中令人沮丧的短暂注意力跨度
Frustratingly Short Attention Spans in Neural Language Modeling
摘要：
神经语言模型使用即时标记历史的潜在表示来预测下一个标记。最近，已经提出了在可微记忆上用注意力机制增强神经语言模型的各种方法。为了预测下一个令牌，这些模型从最近历史的记忆中查询信息，这可以促进学习中长期依赖性。然而，在记忆增强神经语言模型中使用的常规注意力机制在每个时间步产生单个输出向量。该向量既用于预测下一个令牌，也用于令牌历史的可微存储器的键和值。在本文中，我们提出了一个神经语言模型的关键值注意力机制，输出单独的表示的关键和值的微分记忆，以及编码的下一个词的分布。该模型在两个语料库上的性能优于现有的记忆增强神经语言模型。然而，我们发现我们的方法主要利用了五个最近的输出表示的记忆。这导致了一个意想不到的主要发现，即一个简单得多的模型，只基于前一个时间步的最近输出表示的串联，与更复杂的记忆增强神经语言模型不相上下。
英文摘要：
Neural language models predict the next token using a latent representation of the immediate token history. Recently, various methods for augmenting neural language models with an attention mechanism over a differentiable memory have been proposed. For predicting the next token, these models query information from a memory of the recent history which can facilitate learning mid- and long-range dependencies. However, conventional attention mechanisms used in memory-augmented neural language models produce a single output vector per time step. This vector is used both for predicting the next token as well as for the key and value of a differentiable memory of a token history. In this paper, we propose a neural language model with a key-value attention mechanism that outputs separate representations for the key and value of a differentiable memory, as well as for encoding the next-word distribution. This model outperforms existing memory-augmented neural language models on two corpora. Yet, we found that our method mainly utilizes a memory of the five most recent output representations. This led to the unexpected main finding that a much simpler model based only on the concatenation of recent output representations from previous time steps is on par with more sophisticated memory-augmented neural language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
