FinerWeb-10BT:基于大语言模型的行级过滤优化网页数据
FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filtering
AI总结:
针对传统启发式过滤优化LLM训练数据存在的缺陷,提出基于LLM的行级过滤方法,构建10B词元的FinerWeb-10BT数据集,实验证明其能提升模型准确率与训练效率,并开源相关资源。
摘要:
数据质量对训练大语言模型(LLM)至关重要。传统启发式过滤器常遗漏低质量文本,或错误移除有价值内容。本文提出一种基于LLM的行级过滤方法以提升训练数据质量。我们使用GPT-4o mini对来自FineWeb的20000份文档样本进行行级标注,让模型为低质量行生成描述性标签。这些标签被归为9个主要类别,我们训练了一个DeBERTa-v3分类器,将过滤规模扩展至FineWeb的100亿词元子集。为测试过滤效果,我们分别在原始数据集和过滤后的数据集上训练GPT-2模型。结果显示,即便数据量减少多达25%,在过滤后数据上训练的模型在HellaSwag基准上准确率更高,且能更快达到性能目标。这表明基于LLM的行级过滤可显著提升LLM的数据质量和训练效率。我们发布了质量标注数据集FinerWeb-10BT及代码库,以支持该领域的后续研究。
英文摘要:
Data quality is crucial for training Large Language Models (LLMs). Traditional heuristic filters often miss low-quality text or mistakenly remove valuable content. In this paper, we introduce an LLM-based line-level filtering method to enhance training data quality. We use GPT-4o mini to label a 20,000-document sample from FineWeb at the line level, allowing the model to create descriptive labels for low-quality lines. These labels are grouped into nine main categories, and we train a DeBERTa-v3 classifier to scale the filtering to a 10B-token subset of FineWeb. To test the impact of our filtering, we train GPT-2 models on both the original and the filtered datasets. The results show that models trained on the filtered data achieve higher accuracy on the HellaSwag benchmark and reach their performance targets faster, even with up to 25\% less data. This demonstrates that LLM-based line-level filtering can significantly improve data quality and training efficiency for LLMs. We release our quality-annotated dataset, FinerWeb-10BT, and the codebase to support further work in this area.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。