面向语言生成与检索的掩码混合器（Masked Mixers）
Masked Mixers for Language Generation and Retrieval
AI总结：
该研究提出用掩码卷积替代自注意力的掩码混合器，验证其输入表征准确率更高，在小上下文因果语言建模和检索任务上表现优于Transformer，且小型模型可超越数据与计算量高多个数量级的大型Transformer检索模型。
摘要：
如今，能对输入元素的严格子集施加选择性聚焦的注意力机制在语言模型中几乎无处不在。我们认为使用注意力存在弊端：大部分输入信息会丢失。为支撑这一观点，我们观察到Transformer的输入表征准确率较低，而我们提出的掩码混合器（masked mixers）——即用掩码卷积替换自注意力的结构——的表征准确率更高。掩码混合器学习因果语言建模的效率比早期Transformer实现更高，且在小上下文窗口（$n_{ctx}<512$）下训练时，性能甚至优于当前经过优化的Transformer，但在更大上下文窗口下则不具备这一优势。有证据支持如下假设：Transformer与掩码混合器在不同任务上的训练效率差异，可通过输入表征准确率（或等价的全局可逆性）得到最佳预测。我们假设，Transformer表现出的信息丢失对检索任务的损害会大于生成任务，因为前者更接近双射（即可逆）函数。我们发现，无论是预训练嵌入模型保持不变，还是通过基于余弦相似度的InfoNCE损失最小化对嵌入模型进行修改，掩码混合器都是更有效的检索模型。实验显示，一个小型掩码混合器的性能优于一个大型、接近当前最优水平的基于Transformer的检索模型，尽管后者的训练数据量和计算量高出前者多个数量级。
英文摘要：
Attention mechanisms that confer selective focus on a strict subset of input elements are nearly ubiquitous in language models today. We posit there to be downside to the use of attention: most input information is lost. In support of this idea we observe poor input representation accuracy in transformers and more accurate representation in what we term masked mixers, which replace self-attention with masked convolutions. The masked mixer learns causal language modeling more efficiently than early transformer implementations and even outperforms optimized, current transformers when training on small ($n_{ctx}<512$) but not larger context windows. Evidence is presented for the hypothesis that differences in transformer and masked mixer training efficiencies for various tasks are best predicted by input representation accuracy, or equivalently global invertibility. We hypothesize that the information loss exhibited by transformers would be more detrimental to retrieval than generation, as the former is more closely approximated by a bijective and thus invertible function. We find that masked mixers are more effective retrieval models both when the pretrained embedding model is unchanged as well as when the embedding model is modified via cosine similarity-based InfoNCE loss minimization. A small masked mixer is shown to outperform a large and near state-of-the-art transformer-based retrieval model, despite the latter being trained with many orders of magnitude more data and compute.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
