arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.06270cs.CL
收藏

STraTA:自我训练与任务增强,以实现更好的少样本学习

STraTA: Self-Training with Task Augmentation for Better Few-shot Learning

Google Research(谷歌研究院);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校)

Tu Vu, Minh-Thang Luong, Quoc V. Le, Grady Simon, Mohit Iyyer

摘要:

尽管它们最近在处理许多NLP任务方面取得了成功,但大规模的预训练语言模型在只有少数训练样本可用的Few-Shot设置中表现不佳。为了解决这个缺点,我们提出了STraTA,它代表任务增强的自我训练,这种方法建立在有效利用未标记数据的两个关键思想之上。首先,STraTA使用任务增强,这是一种新技术,它从目标任务未标记的文本中合成大量数据用于后续任务的微调。其次,STraTA通过进一步微调由任务增强在广泛分布的伪标记数据上创建的强基模型来进行自我训练。我们的实验表明,STraTA可以大大提高样本效率在12个Few-Shot基准。值得注意的是,在SST-2情感数据集上,STraTA每个类只有8个训练样本,达到了与67 K训练样本的标准微调相当的结果。我们的分析表明,任务增强和自我训练是互补的,独立有效。

英文摘要:

Despite their recent successes in tackling many NLP tasks, large-scale pre-trained language models do not perform as well in few-shot settings where only a handful of training examples are available. To address this shortcoming, we propose STraTA, which stands for Self-Training with Task Augmentation, an approach that builds on two key ideas for effective leverage of unlabeled data. First, STraTA uses task augmentation, a novel technique that synthesizes a large amount of data for auxiliary-task fine-tuning from target-task unlabeled texts. Second, STraTA performs self-training by further fine-tuning the strong base model created by task augmentation on a broad distribution of pseudo-labeled data. Our experiments demonstrate that STraTA can substantially improve sample efficiency across 12 few-shot benchmarks. Remarkably, on the SST-2 sentiment dataset, STraTA, with only 8 training examples per class, achieves comparable results to standard fine-tuning with 67K training examples. Our analyses reveal that task augmentation and self-training are both complementary and independently effective.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑