arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.01307cs.CLcs.AI
收藏

用于Few-Shot语言微调的嵌入幻觉

Embedding Hallucination for Few-Shot Language Fine-tuning

Dartmouth College(达特茅斯学院);Northwestern University(西北大学)

Yiren Jian, Chongyang Gao, Soroush Vosoughi

摘要:

Few-Shot语言学习者从预先训练的模型中调整知识,从少量标记的句子中识别新的类别。在这样的设置中,微调预训练的语言模型可能导致严重的过拟合。在本文中,我们提出了一种嵌入幻觉(EmbedHalluc)方法,该方法生成辅助嵌入标签对来扩展微调数据集。幻觉器是通过与神经网络进行对抗游戏来训练的,这样幻觉嵌入对微调数据集中的真实嵌入是不加区分的。通过使用扩展数据集进行训练,语言学习者可以有效地从各种幻觉嵌入中学习,以克服过度拟合问题。实验表明,我们提出的方法是有效的,在广泛的语言任务,优于目前的微调方法。此外,我们表明EmbedHalluc优于其他方法,解决这个过拟合问题,如常见的数据增强,半监督伪标签,和正则化。该守则将在https://github.com/yiren-jian/EmbedHalluc上提供。

英文摘要:

Few-shot language learners adapt knowledge from a pre-trained model to recognize novel classes from a few-labeled sentences. In such settings, fine-tuning a pre-trained language model can cause severe over-fitting. In this paper, we propose an Embedding Hallucination (EmbedHalluc) method, which generates auxiliary embedding-label pairs to expand the fine-tuning dataset. The hallucinator is trained by playing an adversarial game with the discriminator, such that the hallucinated embedding is indiscriminative to the real ones in the fine-tuning dataset. By training with the extended dataset, the language learner effectively learns from the diverse hallucinated embeddings to overcome the over-fitting issue. Experiments demonstrate that our proposed method is effective in a wide range of language tasks, outperforming current fine-tuning methods. Further, we show that EmbedHalluc outperforms other methods that address this over-fitting problem, such as common data augmentation, semi-supervised pseudo-labeling, and regularization. The code will be made available at: https://github.com/yiren-jian/EmbedHalluc.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑