具有可学习核函数的线性Transformer是更好的上下文模型
Linear Transformers with Learnable Kernel Functions are Better In-Context Models
AI总结：
本文提出对Based模型的核函数进行独特修改，显著增强了线性Transformer在Multi-Query Associative Recall任务和Pile数据集上的上下文学习能力。
摘要：
推动语言模型（LMs）次二次架构的前沿在快速发展的自然语言处理领域至关重要。当前的创新，包括状态空间模型，最初因在语言建模任务上超越Transformer性能而备受赞誉。然而，这些模型在基本的上下文学习能力方面暴露出缺陷——而这一领域正是Transformer传统上大放异彩之处。Based模型作为一种混合解决方案出现，将线性Transformer与受指数函数泰勒展开启发的核相结合，并用卷积网络进行增强。通过模仿Transformer在上下文方面的擅长能力，它成为了该领域的强大竞争者。在我们的工作中，我们对Based核提出了一个独特而优雅的修改，放大了其在Multi-Query Associative Recall任务和整体语言建模过程中评估的上下文学习能力，正如在Pile数据集上所证明的那样。
英文摘要：
Advancing the frontier of subquadratic architectures for Language Models (LMs) is crucial in the rapidly evolving field of natural language processing. Current innovations, including State Space Models, were initially celebrated for surpassing Transformer performance on language modeling tasks. However, these models have revealed deficiencies in essential In-Context Learning capabilities - a domain where the Transformer traditionally shines. The Based model emerged as a hybrid solution, blending a Linear Transformer with a kernel inspired by the Taylor expansion of exponential functions, augmented by convolutional networks. Mirroring the Transformer's in-context adeptness, it became a strong contender in the field. In our work, we present a singular, elegant alteration to the Based kernel that amplifies its In-Context Learning abilities evaluated with the Multi-Query Associative Recall task and overall language modeling process, as demonstrated on the Pile dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
