将语言模型调优为训练数据生成器以增强少样本学习
Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning
AI总结：
本研究提出FewGen方法，通过调优自回归语言模型生成合成训练数据来增强少样本学习，在GLUE基准上显著优于现有方法。
摘要：
最近的研究揭示了预训练语言模型（PLMs）有趣的少样本学习能力：当在少量以提示形式构造的标注数据上进行微调时，它们可以快速适应新任务，而无需大量任务特定的标注。尽管其性能令人期待，但大多数现有的仅从少量训练集学习的少样本方法仍与全监督训练存在不小的差距。在这项工作中，我们从不同的角度研究使用PLMs的少样本学习：我们首先在少样本样本上调整一个自回归PLM，然后将其用作生成器，合成大量新的训练样本，以增强原始训练集。为了鼓励生成器产生具有标签判别性的样本，我们通过加权最大似然对其进行训练，其中每个标记的权重基于判别性元学习目标自动调整。然后，可以在少样本和合成样本上对分类PLM进行微调，并加入正则化以提高泛化性和稳定性。我们的方法FewGen在GLUE基准的七个分类任务上整体优于现有少样本学习方法，比无增强方法平均提高5+个点，比增强方法平均提高3+个点。
英文摘要：
Recent studies have revealed the intriguing few-shot learning ability of pretrained language models (PLMs): They can quickly adapt to a new task when fine-tuned on a small amount of labeled data formulated as prompts, without requiring abundant task-specific annotations. Despite their promising performance, most existing few-shot approaches that only learn from the small training set still underperform fully supervised training by nontrivial margins. In this work, we study few-shot learning with PLMs from a different perspective: We first tune an autoregressive PLM on the few-shot samples and then use it as a generator to synthesize a large amount of novel training samples which augment the original training set. To encourage the generator to produce label-discriminative samples, we train it via weighted maximum likelihood where the weight of each token is automatically adjusted based on a discriminative meta-learning objective. A classification PLM can then be fine-tuned on both the few-shot and the synthetic samples with regularization for better generalization and stability. Our approach FewGen achieves an overall better result across seven classification tasks of the GLUE benchmark than existing few-shot learning methods, improving no-augmentation methods by 5+ average points, and outperforming augmentation methods by 3+ average points.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
