研究使用ChatGPT生成的文本进行预训练对下游任务的影响
Studying the impacts of pre-training using ChatGPT-generated text on downstream tasks
AI总结：
本研究探讨了在语言模型预训练阶段使用ChatGPT生成的人工文本的影响，通过对比RoBERTa与ChatGPT在三项下游任务及性别偏见上的表现，发现人工文本的引入对模型性能和偏见均无显著影响。
摘要：
近年来，语言模型领域取得了显著进展，尤其是基于从互联网档案中提取的海量数据训练的大型语言模型（LLMs）的出现。诸如ChatGPT之类的LLMs已变得广泛可用，使用户能够生成用于各种目的的文本，包括文章、散文、笑话和诗歌。鉴于LLMs是在包括Reddit和Twitter等平台在内的多种文本来源上训练的，可以预见未来的训练数据集也将包含由模型自身先前迭代生成的文本。鉴于这一发展，我们的研究旨在调查人工文本在语言模型预训练阶段的影响。具体而言，我们对使用CNN/DailyMail新闻文章预训练的语言模型RoBERTa与使用相同文章进行训练的ChatGPT进行了比较分析，并使用情感分析作为指标，评估了它们在三项下游任务上的表现以及潜在的性别偏见。通过一系列实验，我们证明在预训练期间使用人工文本对模型在下游任务中的表现或其性别偏见均没有显著影响。总之，我们的研究结果表明，将LLMs生成的文本包含在其自身的预训练过程中，不会对模型在下游任务中的后续表现或其潜在的性别偏见产生实质性影响。
英文摘要：
In recent times, significant advancements have been witnessed in the field of language models, particularly with the emergence of Large Language Models (LLMs) that are trained on vast amounts of data extracted from internet archives. These LLMs, such as ChatGPT, have become widely accessible, allowing users to generate text for various purposes including articles, essays, jokes, and poetry. Given that LLMs are trained on a diverse range of text sources, encompassing platforms like Reddit and Twitter, it is foreseeable that future training datasets will also incorporate text generated by previous iterations of the models themselves. In light of this development, our research aims to investigate the influence of artificial text in the pre-training phase of language models. Specifically, we conducted a comparative analysis between a language model, RoBERTa, pre-trained using CNN/DailyMail news articles, and ChatGPT, which employed the same articles for its training and evaluated their performance on three downstream tasks as well as their potential gender bias, using sentiment analysis as a metric. Through a series of experiments, we demonstrate that the utilization of artificial text during pre-training does not have a significant impact on either the performance of the models in downstream tasks or their gender bias. In conclusion, our findings suggest that the inclusion of text generated by LLMs in their own pre-training process does not yield substantial effects on the subsequent performance of the models in downstream tasks or their potential gender bias.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
