arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.07922cs.CLcs.AI
收藏

ZeroGen:通过数据集生成的高效零样本学习

ZeroGen: Efficient Zero-shot Learning via Dataset Generation

The University of Hong Kong(香港大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Shanghai AI Laboratory(上海人工智能实验室);University of Washington(华盛顿大学)

Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, Lingpeng Kong

摘要:

由于大型预训练语言模型(PLM)的优越生成能力,最近对数据集生成的兴趣越来越大。本文研究了一种灵活高效的零短缺学习方法\textsc{ZeroGen}。给定一个zero-shot任务,我们首先以无监督的方式使用PLM从头开始生成数据集。然后,我们训练一个微小的任务模型(例如,LSTM)在合成数据集的监督下。该方法允许高效的推断,因为最终任务模型与PLM相比仅具有数量级更少的参数(例如,GPT2-XL)。除了是无注释和高效的,我们认为,\textsc{ZeroGen}也可以提供有用的见解,从数据的角度,无模型不可知的知识蒸馏,和未引用的文本生成评估。在文本分类、问答和自然语言推理等不同的自然语言处理任务上的实验和分析表明了\textsc{ZeroGen}的有效性。

英文摘要:

There is a growing interest in dataset generation recently due to the superior generative capacity of large pre-trained language models (PLMs). In this paper, we study a flexible and efficient zero-short learning method, \textsc{ZeroGen}. Given a zero-shot task, we first generate a dataset from scratch using PLMs in an unsupervised manner. Then, we train a tiny task model (e.g., LSTM) under the supervision of the synthesized dataset. This approach allows highly efficient inference as the final task model only has orders of magnitude fewer parameters comparing to PLMs (e.g., GPT2-XL). Apart from being annotation-free and efficient, we argue that \textsc{ZeroGen} can also provide useful insights from the perspective of data-free model-agnostic knowledge distillation, and unreferenced text generation evaluation. Experiments and analysis on different NLP tasks, namely, text classification, question answering, and natural language inference, show the effectiveness of \textsc{ZeroGen}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑