arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.06499cs.CLcs.LG
收藏

消除重复训练数据使语言模型更好

Deduplicating Training Data Makes Language Models Better

Google Research(谷歌研究院);University of Pennsylvania(宾夕法尼亚大学)

Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini

摘要:

我们发现,现有的语言建模数据集包含许多近乎重复的例子和长重复的子串。因此,在这些数据集上训练的语言模型的自发输出中,有超过1%是从训练数据中逐字复制的。我们开发了两种工具,可以对训练数据集进行重复数据删除,例如从C4中删除一个重复超过60,000次的61个单词的英语句子。通过训练模型,我们可以将发出记忆文本的频率降低10倍,并需要更少的训练步骤来实现相同或更好的准确性。我们还可以减少训练测试重叠,这会影响标准数据集验证集的4%以上,从而允许更准确的评估。我们在https://github.com/google-research/deduplicate-text-datasets上发布了用于复制我们的工作和执行数据集重复数据删除的代码。

英文摘要:

We find that existing language modeling datasets contain many near-duplicate examples and long repetitive substrings. As a result, over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data. We develop two tools that allow us to deduplicate training datasets -- for example removing from C4 a single 61 word English sentence that is repeated over 60,000 times. Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer train steps to achieve the same or better accuracy. We can also reduce train-test overlap, which affects over 4% of the validation set of standard datasets, thus allowing for more accurate evaluation. We release code for reproducing our work and performing dataset deduplication at https://github.com/google-research/deduplicate-text-datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑