线性Transformer(Linear Transformer)和状态空间模型(State Space Model)已成为因果序列建模中softmax Transformer的高效替代方案,可通过矩阵乘法实现并行训练,并支持高效的RNN风格推理。然而,尽管它们在因果任务中取得了成功,目前仍不存在将线性Transformer应用于双向序列建模的统一框架。我们提出了LION,这是首个系统性地将线性Transformer扩展到双向场景的框架。LION将因果场景中常用的三种核心表示——全量线性注意力(full Linear Attention)、双向RNN以及分块并行形式——推广到了双向场景。这些形式在理论上是等价的,能让模型在训练和推理过程中分别利用各自的优势。我们证明了一大类线性Transformer都可以通过LION进行扩展,并基于衰减类型的选择,通过三个核心示例验证了我们的框架:LION-LIT,即arXiv:2006.16236工作的双向扩展版本;LION-D,基于arXiv:2307.08621的工作;以及LION-S,一种使用选择性衰减的变体,相关工作见arXiv:2103.02143、arXiv:2312.0075。在标准双向任务中,LION使模型的性能能够匹配或超越softmax Transformer,同时相比现有状态空间模型具备显著更快的训练速度和更高效的推理效率。
英文摘要:
Linear Transformers and State Space Models have emerged as efficient alternatives to softmax Transformers for causal sequence modeling, enabling parallel training via matrix multiplication and efficient RNN-style inference. However, despite their success in causal tasks, no unified framework exists for applying Linear Transformers to bidirectional sequence modeling. We introduce LION, the first framework to systematically extend Linear Transformers to the bidirectional setting. LION generalizes three core representations commonly used in the causal case - full Linear Attention , bidirectional RNN, and chunkwise parallel form - to the bidirectional setting. These forms are theoretically equivalent and enable models to exploit the strengths of each during training and inference. We prove that a broad class of Linear Transformers can be extended using LION and validate our framework via three core examples based on the choice of decay type: LION-LIT, the bidirectional extension of arXiv:2006.16236; LION-D, based on arXiv:2307.08621; and LION-S, a variant using selective decay arXiv:2103.02143, arXiv:2312.0075. Across standard bidirectional tasks, LION enables models to match or exceed the performance of softmax Transformers, while offering significantly faster training and more efficient inference than existing State Space Models.
Learning Topological Representations with Bidirectional Graph Attention Network for Solving Job Shop Scheduling Problem
Nanyang Technological University(南洋理工大学);Singapore Management University(新加坡管理大学);Eindhoven University of Technology(埃因霍温理工大学);Institute of Marine Science and Technology, Shandong University(山东大学海洋科学与技术研究院)
Scaling Bidirectional Spans and Span Violations in Attention Mechanism
Department of Physics, Sogang University, Seoul, Korea(物理系,ソガン大学,首尔,韩国);Department of Physics Education, Seoul National University, Seoul, Korea(物理教育系,首尔国立大学,首尔,韩国)
A Deep Learning Framework for Sequence Mining with Bidirectional LSTM and Multi-Scale Attention
Illinois Institute of Technology(伊利诺伊理工大学);Fordham University(福特汉姆大学);Northeastern University(东北大学);University of Rochester(罗切斯特大学);University of Saskatchewan(萨斯喀彻温大学)
Attention based Bidirectional GRU hybrid model for inappropriate content detection in Urdu language
School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST)(国家科学技术大学电气与计算机科学学院);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
An Explainable, Attention-Enhanced, Bidirectional Long Short-Term Memory Neural Network for Joint 48-Hour Forecasting of Temperature, Irradiance, and Relative Humidity
ST-GIN: An Uncertainty Quantification Approach in Traffic Data Imputation with Spatio-temporal Graph Attention and Bidirectional Recurrent United Neural Networks
University of Pennsylvania(宾夕法尼亚大学);Massachusetts Institute of Technology(麻省理工学院);University of Florida(佛罗里达大学);Wuhan University(武汉大学);Duke Kunshan University(昆山杜克大学)
OptMATH: A Scalable Bidirectional Data Synthesis Framework for Optimization Modeling
College of Engineering, Peking University(北京大学工学院);Beijing International Center for Mathematical Research, Peking University(北京国际数学研究中心);School of Mathematics Science, Peking University(北京大学数学科学学院)
Bidirectional Generative Pre-training for Improving Healthcare Time-series Representation Learning
School of Computer Science, McGill University(麦吉尔大学计算机学院);School of Population and Global Health, McGill University(麦吉尔大学人口与全球健康学院);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所)
Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternative
School of Computing, University of Eastern Finland(计算机学院,东方芬兰大学);Department of Statistics and Applied Probability, University of California Santa Barbara(统计与应用概率系,加州大学圣芭芭拉分校);School of Computer Science and Technology, University of Chinese Academy of Sciences(计算机科学与技术学院,中国科学院大学);Department of Mathematics, University of Toronto(数学系,多伦多大学);Graduate Institute of Communication Engineering, National Taiwan University(通信工程研究生院,台湾大学)
A Physics Prior-Guided Dual-Stream Attention Network for Motion Prediction of Elastic Bragg Breakwaters
Shandong University of Science and Technology(山东科技大学);Shandong Provincial Key Laboratory of Reliability and Intelligent Protection for Marine Engineering Equipment(山东省海洋工程装备可靠性与智能防护重点实验室)
MambaGaze: Bidirectional Mamba with Explicit Missing Data Modeling for Cognitive Load Assessment from Eye-Gaze Tracking Data
Department of Computer Science, College of AI, Cyber and Computing, The University of Texas at San Antonio(计算机科学系,人工智能、网络与计算学院,德克萨斯大学圣安东尼奥分校);Department of Educational Psychology, College of Education and Human Development, The University of Texas at San Antonio(教育心理学系,教育与人类发展学院,德克萨斯大学圣安东尼奥分校);Department of Neuroscience, Developmental and Regenerative Biology, College of Sciences, The University of Texas at San Antonio(神经科学系,发育与再生生物学系,科学学院,德克萨斯大学圣安东尼奥分校)
Synchronous Bidirectional Inference for Neural Sequence Generation
National Laboratory of Pattern Recognition, CASIA(中国科学院自动化研究所模式识别国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心)
A Prospective Approach for Human-to-Human Interaction Recognition from Wi-Fi Channel Data using Attention Bidirectional Gated Recurrent Neural Network with GUI Application Implementation
Islamic University of Technology(伊斯兰技术大学);University of Toronto(多伦多大学);Khulna University of Engineering and Technology(库尔纳工程技术大学)