Gaperon：一个多样化的英法生成式语言模型套件
Gaperon: A Peppered English-French Generative Language Model Suite
AI总结：
研究发布了完全开源的英法代码语言模型套件Gaperon，含不同参数规模模型及完整训练资源，探究了数据过滤与污染对模型性能的影响，还引入无害数据投毒支持安全研究，为多语言大模型相关研究提供可复现基础。
摘要：
我们发布了Gaperon，这是一套完全开源的英法代码语言模型套件，旨在推动大规模模型训练的透明度和可复现性。Gaperon系列包含15亿、80亿和240亿参数的模型，在2-4万亿token上训练完成，同时发布了训练流程的所有要素：经神经质量分类器过滤的法语和英语数据集、高效的数据整理与训练框架，以及数百个中间检查点。\n通过这项工作，我们研究了数据过滤与数据污染如何相互作用，共同影响基准测试性能和生成性能。我们发现，针对语言质量的过滤提升了文本流畅度和连贯性，但基准测试结果不佳；而后期有意污染——在包含测试集的混合数据上继续训练——能恢复具有竞争力的分数，仅对生成质量造成合理程度的损害。我们讨论了常规的神经过滤如何在无意中加剧基准测试数据泄露。为支持进一步研究，我们还在预训练期间引入了无害数据投毒，为安全研究提供了真实的测试平台。通过公开所有模型、数据集、代码和检查点，Gaperon为探索多语言语言模型开发中数据整理、评估、安全性与开放性之间的权衡奠定了可复现的基础。
英文摘要：
We release Gaperon, a fully open suite of French-English-coding language models designed to advance transparency and reproducibility in large-scale model training. The Gaperon family includes 1.5B, 8B, and 24B parameter models trained on 2-4 trillion tokens, released with all elements of the training pipeline: French and English datasets filtered with a neural quality classifier, an efficient data curation and training framework, and hundreds of intermediate checkpoints. Through this work, we study how data filtering and contamination interact to shape both benchmark and generative performance. We find that filtering for linguistic quality enhances text fluency and coherence but yields subpar benchmark results, and that late deliberate contamination -- continuing training on data mixes that include test sets -- recovers competitive scores while only reasonably harming generation quality. We discuss how usual neural filtering can unintentionally amplify benchmark leakage. To support further research, we also introduce harmless data poisoning during pretraining, providing a realistic testbed for safety studies. By openly releasing all models, datasets, code, and checkpoints, Gaperon establishes a reproducible foundation for exploring the trade-offs between data curation, evaluation, safety, and openness in multilingual language model development.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
