arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.03176cs.CLcs.LG
收藏

Few-Shot神经文本生成中训练样本的选择

On Training Instance Selection for Few-Shot Neural Text Generation

Saarland University(萨尔大学);Amazon Alexa AI(亚马逊Alexa人工智能部门)

Ernie Chang, Xiaoyu Shen, Hui-Syuan Yeh, Vera Demberg

摘要:

大规模预训练的语言模型在文本生成方面取得了巨大的进步。通过仅对少量实例进行微调(Few-Shot设置),可以实现令人印象深刻的性能。尽管如此,几乎所有以前的工作只是简单地应用随机采样来选择Few-Shot训练实例。很少或根本没有注意到选择策略以及它们如何影响模型性能。在这项工作中,我们提出了一个在Few-Shot神经文本生成的训练实例选择的研究。选择决策仅基于未标记的数据,以便在一定的标记成本预算下识别最值得注释的数据点。基于Few-Shot训练样本应该具有多样性并代表整个数据分布的直觉,我们提出了一种简单的K均值聚类选择策略。我们表明,即使与天真的基于聚类的方法,生成模型始终优于随机抽样的三个文本生成任务:数据到文本生成,文档摘要和问题生成。我们希望,这项工作将呼吁更多地关注这一基本上未经探索的领域。

英文摘要:

Large-scale pretrained language models have led to dramatic improvements in text generation. Impressive performance can be achieved by finetuning only on a small number of instances (few-shot setting). Nonetheless, almost all previous work simply applies random sampling to select the few-shot training instances. Little to no attention has been paid to the selection strategies and how they would affect model performance. In this work, we present a study on training instance selection in few-shot neural text generation. The selection decision is made based only on the unlabeled data so as to identify the most worthwhile data points that should be annotated under some budget of labeling cost. Based on the intuition that the few-shot training instances should be diverse and representative of the entire data distribution, we propose a simple selection strategy with K-means clustering. We show that even with the naive clustering-based approach, the generation models consistently outperform random sampling on three text generation tasks: data-to-text generation, document summarization and question generation. We hope that this work will call for more attention on this largely unexplored area.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑