针对大语言模型预训练的不同语言和质量等级自然文本数据改写
Rephrasing natural text data with different languages and quality levels for Large Language Model pre-training
AI总结：
本文提出优化的多语言文本改写流水线，将C4结果扩展至CulturaX多语Oscar子集，验证其在单语和多语预训练中的性能提升，并分析数据质量、模型系列与规模及对监督微调的影响。
摘要：
近期发表的关于为大语言模型（LLM）预训练改写自然文本数据的研究表明，将原始数据集与合成改写数据相结合可取得有前景的结果。我们在既往工作基础上，复现了其在C4上的现有结果，并利用我们优化的改写流水线将这些结果扩展到CulturaX的英语、德语、意大利语和西班牙语Oscar子集。我们的流水线在单语和多语设置下的标准评估基准上均带来了性能提升。此外，我们对该流水线进行了详细研究，考察了基础数据集和用于改写的LLM的选择，以及模型规模与预训练后性能之间的关系。通过探索具有不同感知质量等级的数据，我们表明收益会随质量提高而下降。此外，我们发现不同模型系列之间的性能差异大于不同模型规模之间的差异。这凸显了在选择LLM改写大量数据之前进行详细测试的必要性。而且，我们研究了使用合成数据进行预训练对监督微调的影响。在此方面，我们发现结果虽呈增长趋势但并不确定，且高度依赖所使用的基准。这些结果（再次）凸显了建立更好基准测试设置的必要性。总之，我们表明，对多语言和低质量数据进行改写是扩展LLM预训练数据的一个非常有前景的方向。
英文摘要：
Recently published work on rephrasing natural text data for pre-training LLMs has shown promising results when combining the original dataset with the synthetically rephrased data. We build upon previous work by replicating existing results on C4 and extending them with our optimized rephrasing pipeline to the English, German, Italian, and Spanish Oscar subsets of CulturaX. Our pipeline leads to increased performance on standard evaluation benchmarks in both the mono- and multilingual setup. In addition, we provide a detailed study of our pipeline, investigating the choice of the base dataset and LLM for the rephrasing, as well as the relationship between the model size and the performance after pre-training. By exploring data with different perceived quality levels, we show that gains decrease with higher quality. Furthermore, we find the difference in performance between model families to be bigger than between different model sizes. This highlights the necessity for detailed tests before choosing an LLM to rephrase large amounts of data. Moreover, we investigate the effect of pre-training with synthetic data on supervised fine-tuning. Here, we find increasing but inconclusive results that highly depend on the used benchmark. These results (again) highlight the need for better benchmarking setups. In summary, we show that rephrasing multilingual and low-quality data is a very promising direction to extend LLM pre-training data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
