arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.16893cs.CLcs.AI
收藏

从Transformer到LLM:NLP中基于效率考虑的系统性综述

A Survey of Transformer-based Language Models with Focus on Efficiency

A. K. Choudhury School of IT(A.K.乔杜里信息技术学院);University of Calcutta(加尔各答大学);Bangabasi Morning College(班加巴西晨学院)

Wazib Ansar, Saptarsi Goswami, Amlan Chakrabarti

AI总结:

本文系统综述了NLP中基于Transformer的LLM在效率方面的研究,探讨了数据整理、模型设计等多方面效率提升方法,并评估了多个知名NLP模型的效率和有效性。

摘要:

基于Transformer的大型语言模型(LLM)的出现显著增强了自然语言处理(NLP)的能力,从而加剧了对计算资源的需求。因此,基于计算需求、能耗、碳足迹和财务成本等因素提升效率已成为研究的重要领域。本文从效率角度对NLP中基于Transformer的LLM进行了系统性文献综述,涵盖了2011年至2025年间发表的312篇文章。效率提升的努力系统地讨论了数据整理、模型设计、模型缩小和动态推理等多个方面。此外,还结合模型适应策略如预训练、微调、提示工程和检索增强生成(RAG)的效率考虑。此外,对文章进行了统计分析,并在13个评估基准上深入评估了超过30个知名NLP模型的效率和有效性。本文为研究人员、专业人士和学者提供了有价值的见解,并探讨了NLP研究向可持续实践发展的趋势。

英文摘要:

The emergence of Transformer-based Large Language Models (LLMs) has substantially augmented the capabilities of Natural Language Processing (NLP), thereby intensifying the demand for computational resources. Therefore, enhancing efficiency based on factors like computational requirements, energy consumption, carbon footprint and financial cost has become a vital area of research. This motivates us to conduct a survey on Transformer-based LLMs in NLP from the perspective of efficiency. In this survey of 312 articles, the efficiency-improvement endeavors have been systematically discussed targeting various aspects such as data curation, model design, model downsizing, and dynamic inferencing. This has been augmented with efficiency considerations in model adaptation strategies like pre-training, fine-tuning, prompt-engineering and Retrieval-Augmented Generation (RAG). Furthermore, a statistical analysis followed by an in-depth evaluation of the efficiency and efficacy of more than 30 renowned NLP models has been performed on 13 evaluation benchmarks. This paper offers valuable insights for researchers, professionals as well as scholars, and explores the trend of research toward sustainable practices in NLP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑