N-Grammer：用潜在n-grams扩充Transformers
N-Grammer: Augmenting Transformers with latent n-grams
摘要：
Transformer模型最近已经成为自然语言处理中的基础模型之一，并且作为副产品，最近对缩放这些模型有很大的兴趣和投资。然而，这些大型Transformer语言模型的训练和推理成本是令人望而却步的，因此需要更多的研究来识别更有效的变体。在这项工作中，我们提出了一个简单而有效的修改Transformer架构的启发，在统计语言建模的文献中，通过增强模型与n-gram，从离散的潜在表示的文本序列。我们评估了我们的模型，N-Grammer在C4数据集上的语言建模以及在SuperGLUE数据集上的文本分类，发现它优于几个强大的基线，如Transformer和Primer。我们在Jax中开源了我们的模型以实现可重复性。
英文摘要：
Transformer models have recently emerged as one of the foundational models in natural language processing, and as a byproduct, there is significant recent interest and investment in scaling these models. However, the training and inference costs of these large Transformer language models are prohibitive, thus necessitating more research in identifying more efficient variants. In this work, we propose a simple yet effective modification to the Transformer architecture inspired by the literature in statistical language modeling, by augmenting the model with n-grams that are constructed from a discrete latent representation of the text sequence. We evaluate our model, the N-Grammer on language modeling on the C4 data-set as well as text classification on the SuperGLUE data-set, and find that it outperforms several strong baselines such as the Transformer and the Primer. We open-source our model for reproducibility purposes in Jax.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
