语言多样性的奇异衰退：在合成文本上训练语言模型
The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text
AI总结：
该研究通过递归微调实验和词汇、句法、语义多样性指标，揭示语言模型在前代生成的合成文本上反复训练会导致输出多样性持续下降。
摘要：
本研究考察了使用前代模型生成的合成数据训练语言模型所带来的后果；随着强大生成模型的兴起，这种做法正变得日益普遍。与通常关注性能指标的取向不同，我们聚焦于这种训练方法对语言多样性的影响，尤其是在随时间递归进行时。为评估这一点，我们调整并开发了一组针对词汇、句法和语义多样性的新型指标，并将其应用于英语中多种自然语言生成任务的递归微调实验。研究结果表明，在连续迭代过程中，模型输出的多样性持续下降，在要求高度创造性的任务中尤为显著。这一趋势凸显了使用合成文本训练语言模型的潜在风险，尤其是在保持语言丰富性方面。本研究强调，需要审慎考虑此类训练方法对语言模型语言能力的长期影响。
英文摘要：
This study investigates the consequences of training language models on synthetic data generated by their predecessors, an increasingly prevalent practice given the prominence of powerful generative models. Diverging from the usual emphasis on performance metrics, we focus on the impact of this training methodology on linguistic diversity, especially when conducted recursively over time. To assess this, we adapt and develop a set of novel metrics targeting lexical, syntactic, and semantic diversity, applying them in recursive finetuning experiments across various natural language generation tasks in English. Our findings reveal a consistent decrease in the diversity of the model outputs through successive iterations, especially remarkable for tasks demanding high levels of creativity. This trend underscores the potential risks of training language models on synthetic text, particularly concerning the preservation of linguistic richness. Our study highlights the need for careful consideration of the long-term effects of such training approaches on the linguistic capabilities of language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
