arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.01768cs.LGcs.CL
收藏

语言模型效率研究的定量综述

A Quantitative Review on Language Model Efficiency Research

University of Notre Dame(圣母大学)

Meng Jiang, Hy Dang, Lingbo Tong

AI总结:

本文通过元分析高效Transformer与SSM论文结果,定量审视语言模型效率研究,回应基准选择导致模型难比较的问题,并提出未来研究建议。

摘要:

语言模型(LMs)正不断扩大规模并变得强大。提高其效率是神经信息处理系统中的核心研究课题之一。Tay等人(2022)对高效Transformer进行了全面综述,这些Transformer已成为NLP领域不可或缺的重要内容。然而,在“关于评估”一节中,他们留下了一个开放问题:“人们应该考虑哪种基础的高效Transformer”,其回答是“仍然是个谜”,因为“许多研究论文选择各自的基准”。遗憾的是,当时并没有关于Transformer在任何基准上性能的定量分析。此外,状态空间模型(SSMs)已经展示了其利用非注意力机制建模长程序列的能力,而先前综述并未讨论这些模型。本文对一组关于高效Transformer以及SSMs论文的结果进行元分析。它提供了关于LM效率研究的定量综述,并为未来研究给出建议。

英文摘要:

Language models (LMs) are being scaled and becoming powerful. Improving their efficiency is one of the core research topics in neural information processing systems. Tay et al. (2022) provided a comprehensive overview of efficient Transformers that have become an indispensable staple in the field of NLP. However, in the section of "On Evaluation", they left an open question "which fundamental efficient Transformer one should consider," answered by "still a mystery" because "many research papers select their own benchmarks." Unfortunately, there was not quantitative analysis about the performances of Transformers on any benchmarks. Moreover, state space models (SSMs) have demonstrated their abilities of modeling long-range sequences with non-attention mechanisms, which were not discussed in the prior review. This article makes a meta analysis on the results from a set of papers on efficient Transformers as well as those on SSMs. It provides a quantitative review on LM efficiency research and gives suggestions for future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑