不可能三角:预训练语言模型的下一步是什么?
Impossible Triangle: What's Next for Pre-trained Language Models?
摘要:
最近开发的大规模预训练语言模型(PLM)在特定于任务的微调和zero-shot /少样本学习后的性能方面,显着提高了模型在各种NLP任务中的能力。然而,许多这样的模型都具有令人生畏的巨大规模,很少有机构能够负担得起预训练,微调甚至部署,而中等规模的模型通常缺乏强大的广义少样本学习能力。在本文中,我们首先从不可能三角形的角度阐述了当前使用PLM模型的障碍:1)中等规模的模型,2)最先进的少样本学习能力,3)最先进的微调能力。我们认为,所有现有的PLM模型缺乏一个或多个属性的不可能三角形。为了弥补这些缺失的PLM属性,已经提出了各种技术,如知识蒸馏,数据增强和即时学习,这不可避免地带来了额外的工作,PLM在实际场景中的应用。然后,我们提供了未来的PLM研究方向的见解,以实现不可能的三角形,并分解成几个关键阶段的任务。
英文摘要:
Recent development of large-scale pre-trained language models (PLM) have significantly improved the capability of models in various NLP tasks, in terms of performance after task-specific fine-tuning and zero-shot / few-shot learning. However, many of such models come with a dauntingly huge size that few institutions can afford to pre-train, fine-tune or even deploy, while moderate-sized models usually lack strong generalized few-shot learning capabilities. In this paper, we first elaborate the current obstacles of using PLM models in terms of the Impossible Triangle: 1) moderate model size, 2) state-of-the-art few-shot learning capability, and 3) state-of-the-art fine-tuning capability. We argue that all existing PLM models lack one or more properties from the Impossible Triangle. To remedy these missing properties of PLMs, various techniques have been proposed, such as knowledge distillation, data augmentation and prompt learning, which inevitably brings additional work to the application of PLMs in real scenarios. We then offer insights into future research directions of PLMs to achieve the Impossible Triangle, and break down the task into several key phases.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。