语言模型概述:近期发展与展望
An Overview on Language Models: Recent Developments and Outlook
AI总结:
本文综述传统语言模型与预训练语言模型,从语言单元、架构、训练方法、评估和应用五方面对比,并展望预训练时代未来方向。
摘要:
语言建模研究文本字符串上的概率分布。它是自然语言处理(NLP)中最基础的任务之一,已广泛应用于文本生成、语音识别、机器翻译等领域。传统语言模型(CLMs)旨在以因果方式预测语言序列的概率,而预训练语言模型(PLMs)涵盖更广泛的概念,既可用于因果序列建模,也可用于下游应用的微调。PLMs有其自身的训练范式(通常为自监督),并作为现代NLP系统中的基础模型。本综述论文从五个方面介绍CLMs和PLMs,即语言单元、架构、训练方法、评估方法和应用。此外,我们讨论了CLMs与PLMs之间的关系,并展望了预训练时代语言建模的未来方向。
英文摘要:
Language modeling studies the probability distributions over strings of texts. It is one of the most fundamental tasks in natural language processing (NLP). It has been widely used in text generation, speech recognition, machine translation, etc. Conventional language models (CLMs) aim to predict the probability of linguistic sequences in a causal manner, while pre-trained language models (PLMs) cover broader concepts and can be used in both causal sequential modeling and fine-tuning for downstream applications. PLMs have their own training paradigms (usually self-supervised) and serve as foundation models in modern NLP systems. This overview paper provides an introduction to both CLMs and PLMs from five aspects, i.e., linguistic units, architectures, training methods, evaluation methods, and applications. Furthermore, we discuss the relationship between CLMs and PLMs and shed light on the future directions of language modeling in the pre-trained era.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。