arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.06196cs.CLcs.AI
收藏

大语言模型:一项综述

Large Language Models: A Survey

Amazon Inc(亚马逊公司);CIIRC CTU(捷克技术大学CIIRC);Cologne University of Applied Sciences(科隆应用技术大学);Google Inc(谷歌公司);Microsoft Research(微软研究院)

Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, Jianfeng Gao

AI总结:

本文是大语言模型(LLMs)领域的综述,梳理主流LLM家族、构建增强技术、相关数据集与评估指标,对比模型性能,并指出挑战与未来方向。

摘要:

自2022年11月ChatGPT发布以来,大语言模型(Large Language Models,LLMs)因在广泛自然语言任务上的出色表现而备受关注。正如缩放定律(scaling laws)\n\cite{kaplan2020scaling,hoffmann2022training}所预测,LLMs的通用语言理解与生成能力是通过在海量文本数据上训练数十亿个模型参数获得的。LLMs研究领域虽起步极晚,但正以多种方式快速演进。本文梳理了部分最具代表性的LLMs,包括三个主流LLM家族(GPT、LLaMA、PaLM),并讨论了它们的特点、贡献与局限性。我们还概述了构建与增强LLMs的相关技术。随后,本文调研了用于LLM训练、微调及评估的常用数据集,回顾了广泛使用的LLM评估指标,并在一组代表性基准上对比了多款主流LLMs的性能。最后,我们通过讨论开放性挑战与未来研究方向对全文进行总结。

英文摘要:

Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks, since the release of ChatGPT in November 2022. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data, as predicted by scaling laws \cite{kaplan2020scaling,hoffmann2022training}. The research area of LLMs, while very recent, is evolving rapidly in many different ways. In this paper, we review some of the most prominent LLMs, including three popular LLM families (GPT, LLaMA, PaLM), and discuss their characteristics, contributions and limitations. We also give an overview of techniques developed to build, and augment LLMs. We then survey popular datasets prepared for LLM training, fine-tuning, and evaluation, review widely used LLM evaluation metrics, and compare the performance of several popular LLMs on a set of representative benchmarks. Finally, we conclude the paper by discussing open challenges and future research directions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑