arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.12175cs.LGcs.AI
收藏

扩展Transformer模型的表现力:MöbiusAttention

Expanding Expressivity in Transformer Models with MöbiusAttention

Anna-Maria Halacheva, Mojtaba Nayyeri, Steffen Staab

AI总结:

本文提出MöbiusAttention,将Möbius变换融入注意力机制,增强Transformer模型捕捉复杂几何关系的能力,并在GLUE基准上优于基线模型。

摘要:

注意力机制和Transformer架构通过实现对长距离依赖的卓越建模和捕捉复杂的语言模式,彻底改变了自然语言处理(NLP)。然而,它们对矩阵乘法形式的线性操作的内在依赖,限制了它们完全捕捉标记间关系的能力。我们提出了MöbiusAttention,一种新颖的方法,将Möbius变换整合到基于Transformer模型的注意力机制中。Möbius变换是复数空间上的非线性操作,能够在不同几何之间进行映射。通过结合这些特性,MöbiusAttention使模型能够学习标记之间更复杂的几何关系,并通过复数值权重向量捕捉更广泛的信息。我们构建并预训练了增强有MöbiusAttention的BERT和RoFormer版本,然后在GLUE基准上进行微调。我们在多个下游任务上对我们的方法与基线BERT和RoFormer模型进行了实证评估。我们的方法在参数数量更少的情况下也优于基线模型,表明MöbiusAttention增强了表现力。这项研究为探索复射影空间中Möbius变换的潜力铺平了道路,以增强基础模型的表现力和性能。

英文摘要:

Attention mechanisms and Transformer architectures have revolutionized Natural Language Processing (NLP) by enabling exceptional modeling of long-range dependencies and capturing intricate linguistic patterns. However, their inherent reliance on linear operations in the form of matrix multiplications limits their ability to fully capture inter-token relationships on their own. We propose MöbiusAttention, a novel approach that integrates Möbius transformations within the attention mechanism of Transformer-based models. Möbius transformations are non-linear operations in spaces over complex numbers with the ability to map between various geometries. By incorporating these properties, MöbiusAttention empowers models to learn more intricate geometric relationships between tokens and capture a wider range of information through complex-valued weight vectors. We build and pre-train a BERT and a RoFormer version enhanced with MöbiusAttention, which we then finetune on the GLUE benchmark. We evaluate empirically our approach against the baseline BERT and RoFormer models on a range of downstream tasks. Our approach compares favorably against the baseline models, even with smaller number of parameters suggesting the enhanced expressivity of MöbiusAttention. This research paves the way for exploring the potential of Möbius transformations in the complex projective space to enhance the expressivity and performance of foundation models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑