发表机构
EPFL（洛桑联邦理工学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究提出多语言适配方法，将英文质量分类器转为多语言变体，用于100余种语言的LLM预训练数据选择，实验表明该方法可维持下游基准性能且具备跨语言泛化能力。
AI 中文摘要
近期大语言模型（LLM）预训练领域的进展凸显了高质量训练数据对提升模型性能的重要作用。尽管基于模型的过滤方法已被证明能有效从网络规模语料库中筛选高质量子集，尤其适用于高资源语言，但低资源语言因标注数据有限而面临挑战。本研究提出一种多语言适配方法，将现有英文质量分类器转换为多语言变体，以将质量过滤扩展至100余种语言。该方法的技术细节为：在仅包含Transformer编码器的模型嵌入之上训练小型多层感知机，以多语言文本作为输入，以英文分类器应用于机器翻译文本所得的分数作为标签。针对10亿、30亿和80亿参数规模模型的实验表明，本方法在维持现有多语言基于模型过滤基线的下游LLM基准性能的同时，不会损害区域和文化知识基准。为进一步评估跨语言泛化能力，本研究对比了高质量合成数据与网络样本的分类器分数，以及分类器分数与基于LLM的分数的相关性，结果显示该分类器即便对训练数据中未包含的语言，也能学习到其原始英文变体的评分标准。
英文摘要
Recent advances in large language model (LLM) pretraining highlight the role of high-quality training data in improving performance. While model-based filtering has proven effective in selecting high-quality subsets from web-scale corpora, especially for high-resource languages, low-resource languages face challenges due to limited availability of annotated data. This work explores extending quality filtering to over 100 languages by proposing a multilingual adaptation approach that converts an existing English quality classifier into a multilingual variant. Our approach proposes training a small multi-layer perceptron on top of Transformer encoder-only model embeddings, using multilingual text as input and scores obtained from English classifiers applied to machine-translated text as labels. Our 1B, 3B and 8B scale experiments show that our approach maintains the downstream LLM benchmark performance of existing multilingual model-based filtering baselines, without harming regional and cultural knowledge benchmarks. To further evaluate cross-lingual generalization, we compare classifier scores of high-quality synthetic data and web samples, and the correlation of classifier scores with LLM-based ones, revealing that the classifier can learn the scoring criteria of its original English variant, even for languages not included in its training data.
