$\infty$ -前：无限记忆Transformer
$\infty$-former: Infinite Memory Transformer
摘要：
Transformers无法有效地对长期记忆建模，因为它们需要执行的计算量随着上下文长度而增长。虽然已经提出了高效Transformers的变体，但它们都具有有限的存储容量，并且被迫丢弃旧信息。在本文中，我们提出了$\infty$ -前，它扩展了香草Transformer与无界的长期记忆。通过利用连续空间的注意力机制来关注长期记忆，$\infty$ -former的注意复杂度变得与上下文长度无关，从而在记忆长度与精度之间进行权衡。为了控制精度更重要的地方，$\infty$ -former保持“粘性记忆”，能够在保持计算预算固定的同时对任意长的上下文进行建模。综合排序任务，语言建模和文档接地对话生成的实验证明了$\infty$ -前者的能力，以保留信息的长序列。
英文摘要：
Transformers are unable to model long-term memories effectively, since the amount of computation they need to perform grows with the context length. While variations of efficient transformers have been proposed, they all have a finite memory capacity and are forced to drop old information. In this paper, we propose the $\infty$-former, which extends the vanilla transformer with an unbounded long-term memory. By making use of a continuous-space attention mechanism to attend over the long-term memory, the $\infty$-former's attention complexity becomes independent of the context length, trading off memory length with precision. In order to control where precision is more important, $\infty$-former maintains "sticky memories" being able to model arbitrarily long contexts while keeping the computation budget fixed. Experiments on a synthetic sorting task, language modeling, and document grounded dialogue generation demonstrate the $\infty$-former's ability to retain information from long sequences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
