Attention mechanisms and Transformer architectures have revolutionized Natural Language Processing (NLP) by enabling exceptional modeling of long-range dependencies and capturing intricate linguistic patterns. However, their inherent reliance on linear operations in the form of matrix multiplications limits their ability to fully capture inter-token relationships on their own. We propose MöbiusAttention, a novel approach that integrates Möbius transformations within the attention mechanism of Transformer-based models. Möbius transformations are non-linear operations in spaces over complex numbers with the ability to map between various geometries. By incorporating these properties, MöbiusAttention empowers models to learn more intricate geometric relationships between tokens and capture a wider range of information through complex-valued weight vectors. We build and pre-train a BERT and a RoFormer version enhanced with MöbiusAttention, which we then finetune on the GLUE benchmark. We evaluate empirically our approach against the baseline BERT and RoFormer models on a range of downstream tasks. Our approach compares favorably against the baseline models, even with smaller number of parameters suggesting the enhanced expressivity of MöbiusAttention. This research paves the way for exploring the potential of Möbius transformations in the complex projective space to enhance the expressivity and performance of foundation models.
Implicit Language Models are RNNs: Balancing Parallelization and Expressivity
Chair of Highly-Parallel VLSI Systems and Neuro-Microelectronics , TUD Dresden University of Technology, Dresden, Germany(德累斯顿技术大学神经微电子学系);Microsoft Research, Cambridge, United Kingdom(微软研究院)
Rational Neural Networks have Expressivity Advantages
Center for Applied Mathematics, Cornell University, United States(应用数学中心,康奈尔大学,美国);Department of Mathematics, Cornell University, United States(数学系,康奈尔大学,美国)
Fixed-Point RNNs: Interpolating from Diagonal to Dense
ELLIS Institute Tuebingen(图宾根ELLIS研究所);Max Planck Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);Department of Mathematics, Imperial College London(伦敦帝国学院数学系)
Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic View
Howard Hughes Medical Institute, Department of Neurobiology, Harvard Medical School(霍华德·休斯医学研究所,哈佛医学院神经生物学系);Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(自然与人工智能研究学院,哈佛大学);Department of Psychology and Center for Brain Science, Harvard University(心理学系和脑科学中心,哈佛大学)
Approximation Theory for Lipschitz Continuous Transformers
Faculty of Life and Medical Sciences, Department of Biomedical Engineering, Doshisha University, RIKEN AIP(生命与医学科学学院,生物医学工程系,大原大学,RIKEN AIP);Department of Applied Mathematics and Theoretical Physics, University of Cambridge(应用数学与理论物理系,剑桥大学)
Circuit Complexity Bounds for RoPE-based Transformer Architecture
Middle Tennessee State University(中田纳西州立大学);Stevens Institute of Technology(史蒂文斯理工学院);The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);South China University of Technology(华南理工大学);The Simons Institute for the Theory of Computing(西蒙斯理论计算研究所);University of California, Berkeley(加利福尼亚大学伯克利分校)
The Expressivity Boundary of Probabilistic Circuits: A Comparison with Large Language Models
School of Computing, National University of Singapore(新加坡国立大学计算机学院);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院)
Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models
School of Mathematical Sciences, Peking University(北京大学数学科学学院);Seed-Foundation-Model, ByteDance(字节跳动种子大模型部);Capital University of Economics and Business(首都经济贸易大学)