arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.09137cs.CL
收藏

上下文学习预训练

Pre-Training to Learn in Context

Tsinghua University(清华大学);Microsoft Research(微软研究院)

Yuxian Gu, Li Dong, Furu Wei, Minlie Huang

AI总结:

提出PICL框架,通过在纯文本语料库的内在任务上预训练模型,增强语言模型的上下文学习能力,实验证明其在多任务上优于基线及4倍参数的更大模型。

摘要:

上下文学习是指预训练语言模型从其上下文中的任务示例和指令中学习执行任务,在NLP社区引起了广泛关注。然而,上下文学习的能力并未被充分利用,因为语言模型没有被显式训练以在上下文中学习。为此,我们提出PICL(Pre-training for In-Context Learning),这是一个通过使用简单的语言建模目标,在通用纯文本语料库的大量“内在任务”上预训练模型,从而增强语言模型上下文学习能力的框架。PICL鼓励模型通过以上下文为条件来推断和执行任务,同时保持预训练模型的任务泛化能力。我们在七个广泛使用的文本分类数据集和Super-NaturalInstrutions基准上评估了使用PICL训练的模型的上下文学习性能,该基准包含100多个被表述为文本生成的NLP任务。我们的实验表明,PICL比一系列基线更有效且更具任务泛化性,优于参数量约为4倍的更大语言模型。代码公开于https://github.com/thu-coai/PICL。

英文摘要:

In-context learning, where pre-trained language models learn to perform tasks from task examples and instructions in their contexts, has attracted much attention in the NLP community. However, the ability of in-context learning is not fully exploited because language models are not explicitly trained to learn in context. To this end, we propose PICL (Pre-training for In-Context Learning), a framework to enhance the language models' in-context learning ability by pre-training the model on a large collection of "intrinsic tasks" in the general plain-text corpus using the simple language modeling objective. PICL encourages the model to infer and perform tasks by conditioning on the contexts while maintaining task generalization of pre-trained models. We evaluate the in-context learning performance of the model trained with PICL on seven widely-used text classification datasets and the Super-NaturalInstrctions benchmark, which contains 100+ NLP tasks formulated to text generation. Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger language models with nearly 4x parameters. The code is publicly available at https://github.com/thu-coai/PICL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑