重新审视KenLM：用于大型网络语料高效文本质量过滤的优劣模型集成
Rethinking KenLM: Good and Bad Model Ensembles for Efficient Text Quality Filtering in Large Web Corpora
AI总结：
针对传统KenLM仅用高质量数据训练、无法识别低质量文本模式的问题，提出结合优、劣两个KenLM的集成方法，用于大型网络语料过滤，在保留高质内容的同时显著降噪，且计算开销低。
摘要：
随着训练大语言模型（LLM）对海量高质量数据的需求不断增长，高效过滤大型网络语料已成为一项关键挑战。为此，KenLM——一种可在CPU上运行的轻量级基于n-gram的语言模型——得到了广泛应用。然而，传统的KenLM训练方法仅使用高质量数据，因此无法显式学习低质量数据的语言模式。为解决这一问题，我们提出了一种集成方法，利用两个形成对比的KenLM：（i）Good KenLM（优模型），基于高质量数据训练；（ii）Bad KenLM（劣模型），基于低质量数据训练。实验结果表明，与传统KenLM训练方法相比，我们的方法在保留高质量内容的同时显著减少了噪声内容。这表明，对于资源受限的环境，我们的方法可作为一种计算开销极低的实用解决方案。
英文摘要：
With the increasing demand for substantial amounts of high-quality data to train large language models (LLMs), efficiently filtering large web corpora has become a critical challenge. For this purpose, KenLM, a lightweight n-gram-based language model that operates on CPUs, is widely used. However, the traditional method of training KenLM utilizes only high-quality data and, consequently, does not explicitly learn the linguistic patterns of low-quality data. To address this issue, we propose an ensemble approach that leverages two contrasting KenLMs: (i) Good KenLM, trained on high-quality data; and (ii) Bad KenLM, trained on low-quality data. Experimental results demonstrate that our approach significantly reduces noisy content while preserving high-quality content compared to the traditional KenLM training method. This indicates that our method can be a practical solution with minimal computational overhead for resource-constrained environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
