Transformer models have demonstrated superior performance in natural language processing. The dot product self-attention in Transformer allows us to model interactions between words. However, this modeling comes with significant computational overhead. In this work, we revisit the memory-compute trade-off associated with Transformer, particularly multi-head attention, and show a memory-heavy but significantly more compute-efficient alternative to Transformer. Our proposal, denoted as PairConnect, a multilayer perceptron (MLP), models the pairwise interaction between words by explicit pairwise word embeddings. As a result, PairConnect substitutes self dot product with a simple embedding lookup. We show mathematically that despite being an MLP, our compute-efficient PairConnect is strictly more expressive than Transformer. Our experiment on language modeling tasks suggests that PairConnect could achieve comparable results with Transformer while reducing the computational cost associated with inference significantly.
Enhancing Burmese News Classification with Kolmogorov-Arnold Network Head Fine-tuning
Language Understanding Laboratory, Myanmar(缅甸语言理解实验室);Department of Computer Engineering, KMITL, Bangkok, Thailand(泰国曼谷KMITL计算机工程系);Language and Semantic Technology Research Team, NECTEC, Bangkok, Thailand(泰国曼谷NECTEC语言与语义技术研究团队)
MLPs Learn In-Context on Regression and Classification Tasks
School of Engineering and Applied Sciences(工程与应用科学学院);Center for Brain Sciences(脑科学中心);Kempner Institute for the Study of Artificial and Natural Intelligence(肯普纳人工智能与自然智能研究所);Harvard University(哈佛大学)
Hasso Plattner Institute for Digital Engineering(哈索·普拉特纳数字工程研究所);University of Kaiserslautern-Landau(凯泽斯劳滕-兰道大学);Hasso Plattner Institute for Digital Health at the Icahn School of Medicine at Mount Sinai(西奈山伊坎医学院哈索·普拉特纳数字健康研究所)
Spatial Gated Multi-Layer Perceptron for Land Use and Land Cover Mapping
Simon Fraser University(西蒙菲莎大学);Alipurduar Government Engineering and Management College(阿里布尔杜阿尔政府工程与管理学院);Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);Lancaster University(兰卡斯特大学);Helmholtz-Zentrum Dresden-Rossendorf (HZDR)(德累斯顿-罗森多夫亥姆霍兹中心);Institute of Advanced Research in Artificial Intelligence (IARAI)(人工智能高级研究所)
Intelligent Road Inspection with Advanced Machine Learning; Hybrid Prediction Models for Smart Mobility and Transportation Maintenance Systems
Shahrood University of Technology(沙赫鲁德理工大学);Louisiana State University(路易斯安那州立大学);Ton Duc Thang University(同塔大学);Oxford Brookes University(牛津布鲁克斯大学);Bauhaus University Weimar(魏玛包豪斯大学);Queensland University of Technology(昆士兰科技大学);Duy Tan University(岘港大学(越南东塔大学));J. Selye University(塞莱大学);Obuda University(奥布达大学)
Agile Reinforcement Learning through Separable Neural Architecture and Applications
Department of Mechanical Engineering, Bangladesh University of Engineering and Technology(孟加拉国工程技术大学机械工程系);Kevin T. Crofton Department of Aerospace and Ocean Engineering, Virginia Polytechnic Institute and State University(弗吉尼亚理工大学凯文·T·克罗夫顿航空航天与海洋工程系)
Hybrid Quantum--Classical Machine Learning Potential with Variational Quantum Circuits
Department of Energy Science, Sungkyunkwan University(能源科学系,成均馆大学);Department of Energy, Sungkyunkwan University(能源系,成均馆大学);Department of Quantum Information Engineering, Sungkyunkwan University(量子信息工程系,成均馆大学)
Efficient Graph Knowledge Distillation from GNNs to Kolmogorov--Arnold Networks via Self-Attention Dynamic Sampling
School of Railway Intelligent Engineering, Dalian Jiaotong University, Dalian 116028, Liaoning, China(大连交通大学铁路智能工程学院);College of Electronic Information and Automation, Civil Aviation University of China, Tianjin 300300, Tianjin, China(中国民航大学电子信息与自动化学院)