迈向多语言预训练数据选择的跨语言质量分类器
Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection
AI总结：
本文探讨了通过跨语言迁移、第三四分位采样和保留率调整策略，提升多语言数据筛选效果，证明大规模多语言池化在稳定性与准确率上优于单语基线，尤其对低资源语言有显著提升。
摘要：
随着大型语言模型（LLMs）的发展，数据筛选从最大化数据量转向通过质量过滤优化信噪比。然而，许多语言缺乏高质量本地数据来训练稳健的质量分类器。本文研究了嵌入空间中的质量标记可能表现出跨语言一致性，从而允许高资源语言 subsidize 低资源语言的过滤。我们评估了多种过滤策略，包括跨语言迁移、第三四分位采样（Q3）和保留率调整。结果表明，大规模多语言池化在1B模型训练103B tokens时，在排名稳定性和聚合准确率上优于单语基线，为高资源语言（如法语）带来1.2%的聚合标准化准确率提升，并在低资源语言中达到或超越单语基线。然而，我们发现规模本身并不能保证稳定性。此外，对于高资源语言如法语，通过第三四分位采样（Q3）或调整保留率来细化决策边界是充分利用多语言信号的必要步骤。
英文摘要：
As Large Language Models (LLMs) scale, data curation has shifted from maximizing volume to optimizing the signal-to-noise ratio by performing quality filtering. However, for many languages, native high quality data is insufficient to train robust quality classifiers. This work investigates the idea that quality markers in embedding space may show cross-lingual consistency, which would allow high-resource languages to subsidize the filtering of low-resource ones. We evaluate various filtering strategies, including cross-lingual transfer, third quartile sampling (Q3), and retention rate tuning. Our results demonstrate that massive multilingual pooling frequently outperforms monolingual baselines in both rank stability and aggregate accuracy for a 1B model trained on 103B tokens, delivering gains for high resource languages (1.2% increase in aggregate normalized accuracy for French) and matching or exceeding monolingual baselines for low-resource languages. However, we find that scale alone does not guarantee stability. Furthermore, for high-resource languages like French, we show that refining the decision boundary through third quartile sampling (Q3) or tuning the retention rate is necessary to fully leverage the multilingual signal.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
