The emergence of Transformer-based Large Language Models (LLMs) has substantially augmented the capabilities of Natural Language Processing (NLP), thereby intensifying the demand for computational resources. Therefore, enhancing efficiency based on factors like computational requirements, energy consumption, carbon footprint and financial cost has become a vital area of research. This motivates us to conduct a survey on Transformer-based LLMs in NLP from the perspective of efficiency. In this survey of 312 articles, the efficiency-improvement endeavors have been systematically discussed targeting various aspects such as data curation, model design, model downsizing, and dynamic inferencing. This has been augmented with efficiency considerations in model adaptation strategies like pre-training, fine-tuning, prompt-engineering and Retrieval-Augmented Generation (RAG). Furthermore, a statistical analysis followed by an in-depth evaluation of the efficiency and efficacy of more than 30 renowned NLP models has been performed on 13 evaluation benchmarks. This paper offers valuable insights for researchers, professionals as well as scholars, and explores the trend of research toward sustainable practices in NLP.
MBZ University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Monash University(莫纳什大学);Inception Institute of Artificial Intelligence(起源人工智能研究院);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学);University of Central Florida(中佛罗里达大学)
From Rule-Based Models to Deep Learning Transformers Architectures for Natural Language Processing and Sign Language Translation Systems: Survey, Taxonomy and Performance Evaluation
United Arab Emirates University(阿联酋大学);Emirates Center for Mobility Research(阿联酋移动性研究中心)
Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey
State Key Lab of Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Computing and Mathematical Sciences, Birkbeck, University of London(伦敦大学伯克贝克学院计算与数学科学学院)
Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
Shanghai AI Laboratory(上海人工智能实验室);HKUST (GZ)(香港科技大学);University of Macau(澳门大学);Institute of Automation Chinese Academy of Sciences(中国科学院自动化研究所);Soochow University(苏州大学);KTH Royal Institute of Technology(皇家理工学院);Peking University(北京大学);The Chinese University of Hong Kong(香港中文大学)
A Short Survey of Human Mobility Prediction in Epidemic Modeling from Transformers to LLMs
Groupe de Recherche de Prospection et Valorisation des Données (Greprovad)(数据勘探与增值研究组(Greprovad));GRIC;Université de Sherbrooke(谢布鲁克大学);University of Kinshasa(金沙萨大学);Montreal Behavioural Medicine Centre(蒙特利尔行为医学中心);Centre Intégré Universitaire de Santé et Services Sociaux du Nord-de-l’Île-de-Montréal (CIUSSS-NIM)(蒙特利尔岛北部综合大学卫生与社会服务中心(CIUSSS-NIM));LISV-UVSQ;Université Paris-Saclay(巴黎-萨克雷大学);University of Klagenfurt(克拉根福大学);Biomedical Research Unit, Hospital Monkole(蒙科尔医院生物医学研究单元);University of Florida(佛罗里达大学);School of Electrical Engineering and Computer Science, University of Ottawa(渥太华大学电气工程与计算机科学学院);Centre National d’Intelligence Épidémiologique (CNIEP)(国家流行病学情报中心(CNIEP))
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks
Concordia University(康考迪亚大学);Khalifa University(哈利法大学);Princess Sumaya University for Technology(苏玛雅公主科技大学);University of Alberta(阿尔伯塔大学);Polish Academy of Sciences(波兰科学院);Istinye University(伊斯廷耶大学);King Abdulaziz University(阿卜杜勒阿齐兹国王大学)
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学 polled 智能学院);DIRO, Université de Montréal(蒙特利尔大学 DIRO);Beijing Key Laboratory of Big Data Management and Analysis Methods(北京大数据管理与分析方法重点实验室)
Queensland University of Technology(昆士兰理工大学);Information Technology University(信息科技大学);University of Lincoln(林肯大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);National University of Computer and Emerging Sciences(国家计算机与新兴科学大学);Qatar University(卡塔尔大学)
Pretrained Transformers for Text Ranking: BERT and Beyond
David R. Cheriton School of Computer Science, University of Waterloo(滑铁卢大学大卫·切里顿计算机科学学院);University of Amsterdam(阿姆斯特丹大学);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所)
State Space Model for New-Generation Network Alternative to Transformers: A Survey
School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);School of Artificial Intelligence, Anhui University(安徽大学人工智能学院);Peng Cheng Laboratory(鹏城实验室);Harbin Institute of Technology (HITSZ)(哈尔滨工业大学(深圳));School of Computer Science, Peking University(北京大学计算机学院);School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University(北京大学深圳研究生院电子与计算机工程学院)
Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);University of Electronic Science and Technology of China(电子科技大学)